🔄 [AUTO-RESUME] Found existing checkpoint: /workspace/checkpoints/forge_500m_pretrain/emergency_step7 (resuming automatically!) [INFO] Logging to /workspace/checkpoints/forge_500m_pretrain/logs/pretrain_20261019_184032.log ======================================================================== FORGE-500M PRETRAINING ======================================================================== Model config : ./configs/forge_500m.json Data dir : /workspace/data/tokenized Output dir : /workspace/checkpoints/forge_500m_pretrain GPUs : 4 Phase 1 tokens : 0B Phase 2 tokens : 15B Phase 3 tokens : 0B Total tokens : 15B Micro batch/GPU : Phase1=4 | Phase2=4 | Phase3=4 Global batch : 1048576 tokens/step (Phase 2) Resume from : /workspace/checkpoints/forge_500m_pretrain/emergency_step7 Budget cap : $400 ======================================================================== W1019 18:40:33.835000 2022 torch/distributed/run.py:982] W1019 18:40:33.835000 2022 torch/distributed/run.py:982] ***************************************** W1019 18:40:33.835000 2022 torch/distributed/run.py:982] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. W1019 18:40:33.835000 2022 torch/distributed/run.py:982] ***************************************** [2026-10-19 18:40:37][INFO][training.gpu_optimizer] Distributed initialized: rank=3/4, local=3 [2026-10-19 18:40:37][INFO][training.gpu_optimizer] Distributed initialized: rank=0/4, local=0 [2026-10-19 18:40:37][INFO][training.gpu_optimizer] Distributed initialized: rank=2/4, local=2 [2026-10-19 18:40:37][INFO][training.gpu_optimizer] Distributed initialized: rank=1/4, local=1 [2026-10-19 18:40:37][INFO][__main__] ====================================================================== [2026-10-19 18:40:37][INFO][__main__] FORGE-500M PRETRAINING [2026-10-19 18:40:37][INFO][__main__] Rank: 0/4 [2026-10-19 18:40:37][INFO][__main__] Device: cuda:0 [2026-10-19 18:40:37][INFO][__main__] Data: /workspace/data/tokenized [2026-10-19 18:40:37][INFO][__main__] Output: /workspace/checkpoints/forge_500m_pretrain [2026-10-19 18:40:37][INFO][__main__] ====================================================================== [2026-10-19 18:40:37][INFO][__main__] Batch tokens — Phase1: 1,048,576 | Phase2: 1,048,576 | Phase3: 1,048,576 [2026-10-19 18:40:37][INFO][__main__] Batch tokens — Phase1: 1,048,576 | Phase2: 1,048,576 | Phase3: 1,048,576 [2026-10-19 18:40:37][INFO][__main__] Loading CRAYON tokenizer (profile=standard)... [2026-10-19 18:40:37][INFO][__main__] Loading CRAYON tokenizer (profile=standard)... [2026-10-19 18:40:37][INFO][__main__] Batch tokens — Phase1: 1,048,576 | Phase2: 1,048,576 | Phase3: 1,048,576 [2026-10-19 18:40:37][INFO][__main__] Loading CRAYON tokenizer (profile=standard)... [2026-10-19 18:40:37][INFO][__main__] Batch tokens — Phase1: 1,048,576 | Phase2: 1,048,576 | Phase3: 1,048,576 [2026-10-19 18:40:37][INFO][__main__] Loading CRAYON tokenizer (profile=standard)... 🚀 TURBO ENGINE LOADED: Intel(R) Xeon(R) 6787P [Turbo/v5.7.15/64K-SA+AVX2/OMP 64Ksets×2ways intcache=0] 🚀 TURBO ENGINE LOADED: Intel(R) Xeon(R) 6787P [Turbo/v5.7.15/64K-SA+AVX2/OMP 64Ksets×2ways intcache=0] [2026-10-19 18:40:37][INFO][crayon.vocab] Turbo engine loaded successfully [2026-10-19 18:40:37][INFO][crayon.vocab] Turbo engine loaded successfully ✅ CPU BACKEND: Compiled C++ Extension (fallback) ✅ CPU BACKEND: Compiled C++ Extension (fallback) Hardware: Intel(R) Xeon(R) 6787P [Standard] Hardware: Intel(R) Xeon(R) 6787P [Standard] ✅ DEVICE READY: CPU (Compiled C++) ✅ DEVICE READY: CPU (Compiled C++) Hardware: Intel(R) Xeon(R) 6787P [Standard] Hardware: Intel(R) Xeon(R) 6787P [Standard] [2026-10-19 18:40:37][INFO][crayon.vocab] 🔵 CPU Engine Active: Intel(R) Xeon(R) 6787P [Standard] [2026-10-19 18:40:37][INFO][crayon.vocab] 🔵 CPU Engine Active: Intel(R) Xeon(R) 6787P [Standard] 🔧 INITIALIZING DEVICE: CPU 🔧 INITIALIZING DEVICE: CPU 🚀 TURBO ENGINE LOADED: Intel(R) Xeon(R) 6787P [Turbo/v5.7.15/64K-SA+AVX2/OMP 64Ksets×2ways intcache=0] [2026-10-19 18:40:37][INFO][crayon.vocab] Turbo engine loaded successfully ✅ CPU BACKEND: Compiled C++ Extension (fallback) Hardware: Intel(R) Xeon(R) 6787P [Standard] ✅ DEVICE READY: CPU (Compiled C++) Hardware: Intel(R) Xeon(R) 6787P [Standard] [2026-10-19 18:40:37][INFO][crayon.vocab] 🔵 CPU Engine Active: Intel(R) Xeon(R) 6787P [Standard] 🔧 INITIALIZING DEVICE: CPU 🚀 TURBO ENGINE LOADED: Intel(R) Xeon(R) 6787P [Turbo/v5.7.15/64K-SA+AVX2/OMP 64Ksets×2ways intcache=0] [2026-10-19 18:40:37][INFO][crayon.vocab] Turbo engine loaded successfully ✅ CPU BACKEND: Compiled C++ Extension (fallback) Hardware: Intel(R) Xeon(R) 6787P [Standard] ✅ DEVICE READY: CPU (Compiled C++) Hardware: Intel(R) Xeon(R) 6787P [Standard] [2026-10-19 18:40:37][INFO][crayon.vocab] 🔵 CPU Engine Active: Intel(R) Xeon(R) 6787P [Standard] 🔧 INITIALIZING DEVICE: CPU [2026-10-19 18:40:37][INFO][tokenizer.crayon_wrapper] ForgeTokenizer initialized: profile=standard, vocab_size=206464 (crayon=206373 + specials=13 + padding=78), n_workers=172, throughput ~24M tok/s (CPU) [2026-10-19 18:40:37][INFO][__main__] Building FORGE-500M model... [2026-10-19 18:40:37][INFO][tokenizer.crayon_wrapper] ForgeTokenizer initialized: profile=standard, vocab_size=206464 (crayon=206373 + specials=13 + padding=78), n_workers=172, throughput ~24M tok/s (CPU) [2026-10-19 18:40:37][INFO][__main__] Building FORGE-500M model... [2026-10-19 18:40:37][INFO][tokenizer.crayon_wrapper] ForgeTokenizer initialized: profile=standard, vocab_size=206464 (crayon=206373 + specials=13 + padding=78), n_workers=172, throughput ~24M tok/s (CPU) [2026-10-19 18:40:37][INFO][__main__] Tokenizer: ForgeTokenizer(profile='standard', vocab_size=206,464, n_workers=172) [2026-10-19 18:40:37][INFO][tokenizer.crayon_wrapper] Tokenizer config saved to /workspace/checkpoints/forge_500m_pretrain/tokenizer [2026-10-19 18:40:37][INFO][__main__] Building FORGE-500M model... [2026-10-19 18:40:37][INFO][tokenizer.crayon_wrapper] ForgeTokenizer initialized: profile=standard, vocab_size=206464 (crayon=206373 + specials=13 + padding=78), n_workers=172, throughput ~24M tok/s (CPU) [2026-10-19 18:40:37][INFO][__main__] Building FORGE-500M model... /usr/local/lib/python3.12/dist-packages/torch/jit/_script.py:1491: FutureWarning: `torch.jit.script` is deprecated. Please switch to `torch.compile` or `torch.export`. warnings.warn( /usr/local/lib/python3.12/dist-packages/torch/jit/_script.py:1491: FutureWarning: `torch.jit.script` is deprecated. Please switch to `torch.compile` or `torch.export`. warnings.warn( /usr/local/lib/python3.12/dist-packages/torch/jit/_script.py:1491: FutureWarning: `torch.jit.script` is deprecated. Please switch to `torch.compile` or `torch.export`. warnings.warn( /usr/local/lib/python3.12/dist-packages/torch/jit/_script.py:1491: FutureWarning: `torch.jit.script` is deprecated. Please switch to `torch.compile` or `torch.export`. warnings.warn( [2026-10-19 18:40:46][INFO][model.forge_model] ForgeModel: 0.500B total parameters [2026-10-19 18:40:46][INFO][model.forge_model] Embedding: 211.4M [2026-10-19 18:40:46][INFO][model.forge_model] N layers: 18 (3 MHA + 15 ARG, 9 HSE + 9 Dense) [2026-10-19 18:40:46][INFO][model.forge_model] ForgeModel: 0.500B total parameters [2026-10-19 18:40:46][INFO][model.forge_model] Embedding: 211.4M [2026-10-19 18:40:46][INFO][model.forge_model] N layers: 18 (3 MHA + 15 ARG, 9 HSE + 9 Dense) [2026-10-19 18:40:46][INFO][model.forge_model] ForgeModel: 0.500B total parameters [2026-10-19 18:40:46][INFO][model.forge_model] Embedding: 211.4M [2026-10-19 18:40:46][INFO][model.forge_model] N layers: 18 (3 MHA + 15 ARG, 9 HSE + 9 Dense) [2026-10-19 18:40:46][INFO][model.forge_model] ForgeModel: 0.500B total parameters [2026-10-19 18:40:46][INFO][model.forge_model] Embedding: 211.4M [2026-10-19 18:40:46][INFO][model.forge_model] N layers: 18 (3 MHA + 15 ARG, 9 HSE + 9 Dense) [2026-10-19 18:40:46][INFO][__main__] Model: 0.500B total params, 0.500B trainable [2026-10-19 18:40:46][WARNING][__main__] ⚠ Parameter count 0.500B is outside the expected [0.8B, 1.1B] range. Check model_config! [2026-10-19 18:40:46][INFO][model.forge_model] Gradient checkpointing enabled [2026-10-19 18:40:46][INFO][__main__] Gradient checkpointing enabled [2026-10-19 18:40:46][INFO][__main__] Resuming from checkpoint: /workspace/checkpoints/forge_500m_pretrain/emergency_step7 [2026-10-19 18:40:46][INFO][model.forge_model] Gradient checkpointing enabled [2026-10-19 18:40:46][INFO][__main__] Gradient checkpointing enabled [2026-10-19 18:40:46][INFO][__main__] Resuming from checkpoint: /workspace/checkpoints/forge_500m_pretrain/emergency_step7 [2026-10-19 18:40:47][INFO][model.forge_model] Gradient checkpointing enabled [2026-10-19 18:40:47][INFO][__main__] Gradient checkpointing enabled [2026-10-19 18:40:47][INFO][__main__] Resuming from checkpoint: /workspace/checkpoints/forge_500m_pretrain/emergency_step7 [2026-10-19 18:40:47][INFO][model.forge_model] Gradient checkpointing enabled [2026-10-19 18:40:47][INFO][__main__] Gradient checkpointing enabled [2026-10-19 18:40:47][INFO][__main__] Resuming from checkpoint: /workspace/checkpoints/forge_500m_pretrain/emergency_step7 [2026-10-19 18:40:47][INFO][__main__] Model weights loaded from checkpoint [2026-10-19 18:40:47][INFO][__main__] Resuming at 0.01B tokens already processed [2026-10-19 18:40:47][INFO][__main__] Phase 1 disabled (phase1_tokens=0) — starting directly on Phase 2. [2026-10-19 18:40:47][INFO][__main__] Building Phase-2 dataloader (seq=2048, full domain mix)... [2026-10-19 18:40:47][INFO][__main__] Model weights loaded from checkpoint [2026-10-19 18:40:47][INFO][__main__] Resuming at 0.01B tokens already processed [2026-10-19 18:40:47][INFO][data.dataset] Raised open file limit (RLIMIT_NOFILE): 1024 → 65536 [2026-10-19 18:40:47][INFO][__main__] Phase 1 disabled (phase1_tokens=0) — starting directly on Phase 2. [2026-10-19 18:40:47][INFO][__main__] Building Phase-2 dataloader (seq=2048, full domain mix)... [2026-10-19 18:40:47][INFO][__main__] Model weights loaded from checkpoint [2026-10-19 18:40:47][INFO][__main__] Resuming at 0.01B tokens already processed [2026-10-19 18:40:47][INFO][data.dataset] Raised open file limit (RLIMIT_NOFILE): 1024 → 65536 [2026-10-19 18:40:47][INFO][__main__] Phase 1 disabled (phase1_tokens=0) — starting directly on Phase 2. [2026-10-19 18:40:47][INFO][__main__] Building Phase-2 dataloader (seq=2048, full domain mix)... [2026-10-19 18:40:47][INFO][__main__] Model weights loaded from checkpoint [2026-10-19 18:40:47][INFO][__main__] Resuming at 0.01B tokens already processed [2026-10-19 18:40:47][INFO][data.dataset] Raised open file limit (RLIMIT_NOFILE): 1024 → 65536 [2026-10-19 18:40:47][INFO][__main__] Phase 1 disabled (phase1_tokens=0) — starting directly on Phase 2. [2026-10-19 18:40:47][INFO][__main__] Building Phase-2 dataloader (seq=2048, full domain mix)... [2026-10-19 18:40:47][INFO][data.dataset] PackedTokenDataset (train): 7,309,985 sequences × 2048 tokens = 14.97B tokens total [2026-10-19 18:40:47][INFO][__main__] Loaded 'fineweb_edu': 7,309,985 sequences [2026-10-19 18:40:48][INFO][data.dataset] PackedTokenDataset (train): 3,719,165 sequences × 2048 tokens = 7.62B tokens total [2026-10-19 18:40:48][INFO][__main__] Loaded 'thestack': 3,719,165 sequences [2026-10-19 18:40:48][INFO][data.dataset] PackedTokenDataset (train): 7,309,985 sequences × 2048 tokens = 14.97B tokens total [2026-10-19 18:40:48][INFO][__main__] Loaded 'fineweb_edu': 7,309,985 sequences [2026-10-19 18:40:48][INFO][data.dataset] PackedTokenDataset (train): 2,088,970 sequences × 2048 tokens = 4.28B tokens total [2026-10-19 18:40:48][INFO][__main__] Loaded 'wikipedia': 2,088,970 sequences [2026-10-19 18:40:48][INFO][data.dataset] PackedTokenDataset (train): 7,309,985 sequences × 2048 tokens = 14.97B tokens total [2026-10-19 18:40:48][INFO][__main__] Loaded 'fineweb_edu': 7,309,985 sequences [2026-10-19 18:40:48][INFO][data.dataset] PackedTokenDataset (train): 1,935,817 sequences × 2048 tokens = 3.96B tokens total [2026-10-19 18:40:48][INFO][__main__] Loaded 'openwebmath': 1,935,817 sequences [2026-10-19 18:40:48][INFO][data.dataset] PackedTokenDataset (train): 7,309,985 sequences × 2048 tokens = 14.97B tokens total [2026-10-19 18:40:48][INFO][__main__] Loaded 'fineweb_edu': 7,309,985 sequences [2026-10-19 18:40:48][INFO][data.dataset] PackedTokenDataset (train): 1,350,980 sequences × 2048 tokens = 2.77B tokens total [2026-10-19 18:40:48][INFO][__main__] Loaded 'books': 1,350,980 sequences [2026-10-19 18:40:48][INFO][data.dataset] PackedTokenDataset (train): 3,719,165 sequences × 2048 tokens = 7.62B tokens total [2026-10-19 18:40:48][INFO][__main__] Loaded 'thestack': 3,719,165 sequences [2026-10-19 18:40:48][INFO][data.dataset] PackedTokenDataset (train): 3,719,165 sequences × 2048 tokens = 7.62B tokens total [2026-10-19 18:40:48][INFO][__main__] Loaded 'thestack': 3,719,165 sequences [2026-10-19 18:40:48][INFO][data.dataset] PackedTokenDataset (train): 3,719,165 sequences × 2048 tokens = 7.62B tokens total [2026-10-19 18:40:48][INFO][__main__] Loaded 'thestack': 3,719,165 sequences [2026-10-19 18:40:48][INFO][data.dataset] PackedTokenDataset (train): 2,088,970 sequences × 2048 tokens = 4.28B tokens total [2026-10-19 18:40:48][INFO][__main__] Loaded 'wikipedia': 2,088,970 sequences [2026-10-19 18:40:48][INFO][data.dataset] PackedTokenDataset (train): 3,202,429 sequences × 2048 tokens = 6.56B tokens total [2026-10-19 18:40:48][INFO][__main__] Loaded 'arxiv': 3,202,429 sequences [2026-10-19 18:40:48][INFO][data.dataset] PackedTokenDataset (train): 2,088,970 sequences × 2048 tokens = 4.28B tokens total [2026-10-19 18:40:48][INFO][__main__] Loaded 'wikipedia': 2,088,970 sequences [2026-10-19 18:40:48][INFO][data.dataset] PackedTokenDataset (train): 1,935,817 sequences × 2048 tokens = 3.96B tokens total [2026-10-19 18:40:48][INFO][__main__] Loaded 'openwebmath': 1,935,817 sequences [2026-10-19 18:40:48][INFO][data.dataset] PackedTokenDataset (train): 2,088,970 sequences × 2048 tokens = 4.28B tokens total [2026-10-19 18:40:48][INFO][__main__] Loaded 'wikipedia': 2,088,970 sequences [2026-10-19 18:40:48][INFO][data.dataset] PackedTokenDataset (train): 2,418,666 sequences × 2048 tokens = 4.95B tokens total [2026-10-19 18:40:48][INFO][__main__] Loaded 'dolma': 2,418,666 sequences [2026-10-19 18:40:48][INFO][data.dataset] PackedTokenDataset (train): 1,350,980 sequences × 2048 tokens = 2.77B tokens total [2026-10-19 18:40:48][INFO][__main__] Loaded 'books': 1,350,980 sequences [2026-10-19 18:40:48][INFO][data.dataset] PackedTokenDataset (train): 1,935,817 sequences × 2048 tokens = 3.96B tokens total [2026-10-19 18:40:48][INFO][__main__] Loaded 'openwebmath': 1,935,817 sequences [2026-10-19 18:40:48][INFO][data.dataset] PackedTokenDataset (train): 1,935,817 sequences × 2048 tokens = 3.96B tokens total [2026-10-19 18:40:48][INFO][__main__] Loaded 'openwebmath': 1,935,817 sequences [2026-10-19 18:40:48][INFO][data.dataset] PackedTokenDataset (train): 1,178,010 sequences × 2048 tokens = 2.41B tokens total [2026-10-19 18:40:48][INFO][__main__] Loaded 'stackexchange': 1,178,010 sequences [2026-10-19 18:40:48][INFO][data.dataset] PackedTokenDataset (train): 520,865 sequences × 2048 tokens = 1.07B tokens total [2026-10-19 18:40:48][INFO][__main__] Loaded 'redpajama_cc': 520,865 sequences [2026-10-19 18:40:48][INFO][data.dataset] PackedTokenDataset (train): 1,350,980 sequences × 2048 tokens = 2.77B tokens total [2026-10-19 18:40:48][INFO][__main__] Loaded 'books': 1,350,980 sequences [2026-10-19 18:40:48][INFO][data.dataset] PackedTokenDataset (train): 973,844 sequences × 2048 tokens = 1.99B tokens total [2026-10-19 18:40:48][INFO][__main__] Loaded 'multilingual': 973,844 sequences [2026-10-19 18:40:48][INFO][data.dataset] PackedTokenDataset (train): 1,350,980 sequences × 2048 tokens = 2.77B tokens total [2026-10-19 18:40:48][INFO][__main__] Loaded 'books': 1,350,980 sequences [2026-10-19 18:40:48][INFO][data.dataset] PackedTokenDataset (train): 3,202,429 sequences × 2048 tokens = 6.56B tokens total [2026-10-19 18:40:48][INFO][__main__] Loaded 'arxiv': 3,202,429 sequences [2026-10-19 18:40:48][INFO][data.dataset] PackedTokenDataset (train): 3,202,429 sequences × 2048 tokens = 6.56B tokens total [2026-10-19 18:40:48][INFO][__main__] Loaded 'arxiv': 3,202,429 sequences [2026-10-19 18:40:48][INFO][data.dataset] PackedTokenDataset (train): 2,418,666 sequences × 2048 tokens = 4.95B tokens total [2026-10-19 18:40:48][INFO][__main__] Loaded 'dolma': 2,418,666 sequences [2026-10-19 18:40:48][INFO][data.dataset] PackedTokenDataset (train): 3,202,429 sequences × 2048 tokens = 6.56B tokens total [2026-10-19 18:40:48][INFO][__main__] Loaded 'arxiv': 3,202,429 sequences [2026-10-19 18:40:48][INFO][data.dataset] PackedTokenDataset (train): 1,178,010 sequences × 2048 tokens = 2.41B tokens total [2026-10-19 18:40:48][INFO][__main__] Loaded 'stackexchange': 1,178,010 sequences [2026-10-19 18:40:48][INFO][data.dataset] PackedTokenDataset (train): 520,865 sequences × 2048 tokens = 1.07B tokens total [2026-10-19 18:40:48][INFO][__main__] Loaded 'redpajama_cc': 520,865 sequences [2026-10-19 18:40:48][INFO][data.dataset] PackedTokenDataset (train): 2,418,666 sequences × 2048 tokens = 4.95B tokens total [2026-10-19 18:40:48][INFO][__main__] Loaded 'dolma': 2,418,666 sequences [2026-10-19 18:40:48][INFO][data.dataset] PackedTokenDataset (train): 973,844 sequences × 2048 tokens = 1.99B tokens total [2026-10-19 18:40:48][INFO][__main__] Loaded 'multilingual': 973,844 sequences [2026-10-19 18:40:48][INFO][data.dataset] PackedTokenDataset (train): 1,178,010 sequences × 2048 tokens = 2.41B tokens total [2026-10-19 18:40:48][INFO][__main__] Loaded 'stackexchange': 1,178,010 sequences [2026-10-19 18:40:48][INFO][data.dataset] PackedTokenDataset (train): 520,865 sequences × 2048 tokens = 1.07B tokens total [2026-10-19 18:40:48][INFO][__main__] Loaded 'redpajama_cc': 520,865 sequences [2026-10-19 18:40:48][INFO][data.dataset] PackedTokenDataset (train): 2,418,666 sequences × 2048 tokens = 4.95B tokens total [2026-10-19 18:40:48][INFO][__main__] Loaded 'dolma': 2,418,666 sequences [2026-10-19 18:40:48][INFO][data.dataset] PackedTokenDataset (train): 973,844 sequences × 2048 tokens = 1.99B tokens total [2026-10-19 18:40:48][INFO][__main__] Loaded 'multilingual': 973,844 sequences [2026-10-19 18:40:48][INFO][data.dataset] WeightedDataMixer: 7,324,218 total samples from 10 sources with weights {'fineweb_edu': 0.3, 'thestack': 0.16, 'wikipedia': 0.08, 'openwebmath': 0.08, 'books': 0.07, 'arxiv': 0.06, 'dolma': 0.1, 'stackexchange': 0.05, 'redpajama_cc': 0.06, 'multilingual': 0.04} [2026-10-19 18:40:48][INFO][__main__] Phase-2 loader ready: 7,324,218 sequences (batch_size=4), grad_accum=32 [2026-10-19 18:40:48][INFO][__main__] Phase 3 disabled (phase3_tokens=0) — pure seq=2048 pretraining. [2026-10-19 18:40:48][INFO][__main__] Building AdamW optimizer with parameter-group differentiation... [2026-10-19 18:40:48][INFO][training.optimizer] Optimizer group 'main': 287.9M params, lr=3.00e-04, wd=0.1 [2026-10-19 18:40:48][INFO][training.optimizer] Optimizer group 'no_decay': 0.1M params, lr=3.00e-04, wd=0.0 [2026-10-19 18:40:48][INFO][training.optimizer] Optimizer group 'embedding': 211.4M params, lr=1.50e-04, wd=0.0 [2026-10-19 18:40:48][INFO][training.optimizer] Optimizer group 'ssm_state': 0.0M params, lr=9.00e-05, wd=0.0 [2026-10-19 18:40:48][INFO][training.optimizer] Optimizer group 'moe_router': 0.3M params, lr=1.50e-04, wd=0.0 [2026-10-19 18:40:48][INFO][training.optimizer] Using fused AdamW (CUDA native) [2026-10-19 18:40:48][INFO][data.dataset] PackedTokenDataset (train): 1,178,010 sequences × 2048 tokens = 2.41B tokens total [2026-10-19 18:40:48][INFO][__main__] Loaded 'stackexchange': 1,178,010 sequences [2026-10-19 18:40:48][INFO][data.dataset] PackedTokenDataset (train): 520,865 sequences × 2048 tokens = 1.07B tokens total [2026-10-19 18:40:48][INFO][__main__] Loaded 'redpajama_cc': 520,865 sequences [2026-10-19 18:40:48][INFO][data.dataset] PackedTokenDataset (train): 973,844 sequences × 2048 tokens = 1.99B tokens total [2026-10-19 18:40:48][INFO][__main__] Loaded 'multilingual': 973,844 sequences [2026-10-19 18:40:48][INFO][data.dataset] WeightedDataMixer: 7,324,218 total samples from 10 sources with weights {'fineweb_edu': 0.3, 'thestack': 0.16, 'wikipedia': 0.08, 'openwebmath': 0.08, 'books': 0.07, 'arxiv': 0.06, 'dolma': 0.1, 'stackexchange': 0.05, 'redpajama_cc': 0.06, 'multilingual': 0.04} [2026-10-19 18:40:48][INFO][__main__] Phase-2 loader ready: 7,324,218 sequences (batch_size=4), grad_accum=32 [2026-10-19 18:40:48][INFO][__main__] Phase 3 disabled (phase3_tokens=0) — pure seq=2048 pretraining. [2026-10-19 18:40:48][INFO][__main__] Building AdamW optimizer with parameter-group differentiation... [2026-10-19 18:40:48][INFO][training.optimizer] Optimizer group 'main': 287.9M params, lr=3.00e-04, wd=0.1 [2026-10-19 18:40:48][INFO][training.optimizer] Optimizer group 'no_decay': 0.1M params, lr=3.00e-04, wd=0.0 [2026-10-19 18:40:48][INFO][training.optimizer] Optimizer group 'embedding': 211.4M params, lr=1.50e-04, wd=0.0 [2026-10-19 18:40:48][INFO][training.optimizer] Optimizer group 'ssm_state': 0.0M params, lr=9.00e-05, wd=0.0 [2026-10-19 18:40:48][INFO][training.optimizer] Optimizer group 'moe_router': 0.3M params, lr=1.50e-04, wd=0.0 [2026-10-19 18:40:48][INFO][training.optimizer] Using fused AdamW (CUDA native) [2026-10-19 18:40:48][INFO][data.dataset] WeightedDataMixer: 7,324,218 total samples from 10 sources with weights {'fineweb_edu': 0.3, 'thestack': 0.16, 'wikipedia': 0.08, 'openwebmath': 0.08, 'books': 0.07, 'arxiv': 0.06, 'dolma': 0.1, 'stackexchange': 0.05, 'redpajama_cc': 0.06, 'multilingual': 0.04} [2026-10-19 18:40:48][INFO][__main__] Phase-2 loader ready: 7,324,218 sequences (batch_size=4), grad_accum=32 [2026-10-19 18:40:48][INFO][__main__] Phase 3 disabled (phase3_tokens=0) — pure seq=2048 pretraining. [2026-10-19 18:40:48][INFO][__main__] Building AdamW optimizer with parameter-group differentiation... [2026-10-19 18:40:48][INFO][training.optimizer] Optimizer group 'main': 287.9M params, lr=3.00e-04, wd=0.1 [2026-10-19 18:40:48][INFO][training.optimizer] Optimizer group 'no_decay': 0.1M params, lr=3.00e-04, wd=0.0 [2026-10-19 18:40:48][INFO][training.optimizer] Optimizer group 'embedding': 211.4M params, lr=1.50e-04, wd=0.0 [2026-10-19 18:40:48][INFO][training.optimizer] Optimizer group 'ssm_state': 0.0M params, lr=9.00e-05, wd=0.0 [2026-10-19 18:40:48][INFO][training.optimizer] Optimizer group 'moe_router': 0.3M params, lr=1.50e-04, wd=0.0 [2026-10-19 18:40:48][INFO][training.optimizer] Using fused AdamW (CUDA native) [2026-10-19 18:40:48][INFO][data.dataset] WeightedDataMixer: 7,324,218 total samples from 10 sources with weights {'fineweb_edu': 0.3, 'thestack': 0.16, 'wikipedia': 0.08, 'openwebmath': 0.08, 'books': 0.07, 'arxiv': 0.06, 'dolma': 0.1, 'stackexchange': 0.05, 'redpajama_cc': 0.06, 'multilingual': 0.04} [2026-10-19 18:40:48][INFO][__main__] Phase-2 loader ready: 7,324,218 sequences (batch_size=4), grad_accum=32 [2026-10-19 18:40:48][INFO][__main__] Phase 3 disabled (phase3_tokens=0) — pure seq=2048 pretraining. [2026-10-19 18:40:48][INFO][__main__] Building AdamW optimizer with parameter-group differentiation... [2026-10-19 18:40:48][INFO][training.optimizer] Optimizer group 'main': 287.9M params, lr=3.00e-04, wd=0.1 [2026-10-19 18:40:48][INFO][training.optimizer] Optimizer group 'no_decay': 0.1M params, lr=3.00e-04, wd=0.0 [2026-10-19 18:40:48][INFO][training.optimizer] Optimizer group 'embedding': 211.4M params, lr=1.50e-04, wd=0.0 [2026-10-19 18:40:48][INFO][training.optimizer] Optimizer group 'ssm_state': 0.0M params, lr=9.00e-05, wd=0.0 [2026-10-19 18:40:48][INFO][training.optimizer] Optimizer group 'moe_router': 0.3M params, lr=1.50e-04, wd=0.0 [2026-10-19 18:40:48][INFO][training.optimizer] Using fused AdamW (CUDA native) [2026-10-19 18:40:49][INFO][__main__] ✅ Optimizer state restored exactly from checkpoint. [2026-10-19 18:40:49][INFO][__main__] ✅ Scheduler state restored exactly from checkpoint. [2026-10-19 18:40:49][INFO][__main__] ✅ Optimizer state restored exactly from checkpoint. [2026-10-19 18:40:49][INFO][__main__] ✅ Scheduler state restored exactly from checkpoint. [2026-10-19 18:40:49][INFO][__main__] ✅ Optimizer state restored exactly from checkpoint. [2026-10-19 18:40:49][INFO][__main__] ✅ Scheduler state restored exactly from checkpoint. [2026-10-19 18:40:49][INFO][__main__] ✅ Optimizer state restored exactly from checkpoint. [2026-10-19 18:40:49][INFO][__main__] ✅ Scheduler state restored exactly from checkpoint. [2026-10-19 18:40:50][INFO][training.gpu_optimizer] GPU Memory[pre-training] step=0: 2.44GB alloc / 2.46GB reserved / 30.29GB free / 32.75GB total [2026-10-19 18:40:50][INFO][training.gpu_optimizer] GPU Memory[pre-training] step=0: 2.44GB alloc / 2.46GB reserved / 30.29GB free / 32.75GB total [2026-10-19 18:40:50][INFO][training.gpu_optimizer] GPU Memory[pre-training] step=0: 2.44GB alloc / 2.46GB reserved / 30.29GB free / 32.75GB total [2026-10-19 18:40:50][INFO][training.gpu_optimizer] GPU Memory[pre-training] step=0: 2.44GB alloc / 2.46GB reserved / 30.29GB free / 32.75GB total [2026-10-19 18:40:50][INFO][__main__] Initializing PretrainEngine... [2026-10-19 18:40:50][INFO][__main__] Initializing PretrainEngine... [2026-10-19 18:40:50][INFO][__main__] Initializing PretrainEngine... [2026-10-19 18:40:50][INFO][__main__] Initializing PretrainEngine... [2026-10-19 18:40:50][INFO][training.gpu_optimizer] GPU detected: NVIDIA B300 SXM6 AC MIG 1g.34gb (MIG 18/144 SMs) — 312.5 TFLOPS BF16 (MIG slice peak) [2026-10-19 18:40:50][INFO][__main__] Engine token counter initialised to 0.01B [2026-10-19 18:40:50][INFO][training.gpu_optimizer] Warming up GPU cuda:1... [2026-10-19 18:40:50][INFO][training.gpu_optimizer] GPU detected: NVIDIA B300 SXM6 AC MIG 1g.34gb (MIG 18/144 SMs) — 312.5 TFLOPS BF16 (MIG slice peak) [2026-10-19 18:40:50][INFO][__main__] Engine token counter initialised to 0.01B [2026-10-19 18:40:50][INFO][training.gpu_optimizer] Warming up GPU cuda:2... [2026-10-19 18:40:50][INFO][training.gpu_optimizer] GPU detected: NVIDIA B300 SXM6 AC MIG 1g.34gb (MIG 18/144 SMs) — 312.5 TFLOPS BF16 (MIG slice peak) [2026-10-19 18:40:50][INFO][training.gpu_optimizer] GPU detected: NVIDIA B300 SXM6 AC MIG 1g.34gb (MIG 18/144 SMs) — 312.5 TFLOPS BF16 (MIG slice peak) [2026-10-19 18:40:50][INFO][__main__] Engine token counter initialised to 0.01B [2026-10-19 18:40:50][INFO][__main__] Engine token counter initialised to 0.01B [2026-10-19 18:40:50][INFO][training.gpu_optimizer] Warming up GPU cuda:3... [2026-10-19 18:40:50][INFO][__main__] ====================================================================== [2026-10-19 18:40:50][INFO][__main__] FORGE-3B PRETRAINING — FULL THREE-PHASE RUN [2026-10-19 18:40:50][INFO][__main__] Phase 1 : 0B tokens | seq=512 [2026-10-19 18:40:50][INFO][__main__] Phase 2 : 15B tokens | seq=2048 [2026-10-19 18:40:50][INFO][__main__] Phase 3 : 0B tokens | seq=2048 [2026-10-19 18:40:50][INFO][__main__] Total : 15B tokens [2026-10-19 18:40:50][INFO][__main__] Budget : $450.00 | Rate: $63.17/hr (16× H100 SXM) [2026-10-19 18:40:50][INFO][__main__] ====================================================================== [2026-10-19 18:40:50][INFO][training.gpu_optimizer] Warming up GPU cuda:0... [2026-10-19 18:40:50][INFO][training.gpu_optimizer] GPU cuda:1 warmed up. Memory: 2.47GB allocated, 2.50GB reserved /workspace/forge_3b/training/pretrain_engine.py:106: FutureWarning: `broadcast_buffers` is deprecated. Use `forward_sync_buffers` instead. IMPORTANT: unlike `broadcast_buffers=False`, `forward_sync_buffers=False` still syncs buffers at init. If you rely on buffers NOT being synced at init (e.g. rank-local buffers), keep using `broadcast_buffers=False` until `init_sync_buffers` is available. self.model = DDP( [2026-10-19 18:40:50][INFO][training.gpu_optimizer] GPU cuda:3 warmed up. Memory: 2.47GB allocated, 2.50GB reserved /workspace/forge_3b/training/pretrain_engine.py:106: FutureWarning: `broadcast_buffers` is deprecated. Use `forward_sync_buffers` instead. IMPORTANT: unlike `broadcast_buffers=False`, `forward_sync_buffers=False` still syncs buffers at init. If you rely on buffers NOT being synced at init (e.g. rank-local buffers), keep using `broadcast_buffers=False` until `init_sync_buffers` is available. self.model = DDP( [2026-10-19 18:40:50][INFO][training.gpu_optimizer] GPU cuda:2 warmed up. Memory: 2.47GB allocated, 2.50GB reserved /workspace/forge_3b/training/pretrain_engine.py:106: FutureWarning: `broadcast_buffers` is deprecated. Use `forward_sync_buffers` instead. IMPORTANT: unlike `broadcast_buffers=False`, `forward_sync_buffers=False` still syncs buffers at init. If you rely on buffers NOT being synced at init (e.g. rank-local buffers), keep using `broadcast_buffers=False` until `init_sync_buffers` is available. self.model = DDP( [2026-10-19 18:40:50][INFO][training.gpu_optimizer] GPU cuda:0 warmed up. Memory: 2.47GB allocated, 2.50GB reserved /workspace/forge_3b/training/pretrain_engine.py:106: FutureWarning: `broadcast_buffers` is deprecated. Use `forward_sync_buffers` instead. IMPORTANT: unlike `broadcast_buffers=False`, `forward_sync_buffers=False` still syncs buffers at init. If you rely on buffers NOT being synced at init (e.g. rank-local buffers), keep using `broadcast_buffers=False` until `init_sync_buffers` is available. self.model = DDP( [2026-10-19 18:40:50][INFO][training.pretrain_engine] 🚀 Wrapped model in DDP (world_size=4, static_graph=False, find_unused_parameters=True) [2026-10-19 18:40:50][INFO][training.pretrain_engine] ====================================================================== [2026-10-19 18:40:50][INFO][training.pretrain_engine] FORGE-500M PRETRAINING STARTED (plain PyTorch DDP, no DeepSpeed) [2026-10-19 18:40:50][INFO][training.pretrain_engine] Target: 15B tokens (Chinchilla-optimal for 500M) [2026-10-19 18:40:50][INFO][training.pretrain_engine] Budget: $400 | Cost rate: $13.16/hr [2026-10-19 18:40:50][INFO][training.pretrain_engine] Max training hours: 30.4h [2026-10-19 18:40:50][INFO][training.pretrain_engine] World size: 4 [2026-10-19 18:40:50][INFO][training.pretrain_engine] ====================================================================== [2026-10-19 18:40:50][INFO][training.pretrain_engine] ================================================== [2026-10-19 18:40:50][INFO][training.pretrain_engine] PHASE 1: SKIPPED (Starting directly on Phase 2: Core Pretraining @ seq=2048) [2026-10-19 18:40:50][INFO][training.pretrain_engine] ARG local window updated to 64 [2026-10-19 18:40:50][INFO][training.pretrain_engine] ================================================== [2026-10-19 18:40:50][INFO][training.pretrain_engine] PHASE 2: Core Pre-Training (15.0B tokens, seq=2048) [2026-10-19 18:40:50][INFO][training.pretrain_engine] 🚀 Wrapped model in DDP (world_size=4, static_graph=False, find_unused_parameters=True) [2026-10-19 18:40:50][INFO][training.pretrain_engine] ====================================================================== [2026-10-19 18:40:50][INFO][training.pretrain_engine] FORGE-500M PRETRAINING STARTED (plain PyTorch DDP, no DeepSpeed) [2026-10-19 18:40:50][INFO][training.pretrain_engine] Target: 15B tokens (Chinchilla-optimal for 500M) [2026-10-19 18:40:50][INFO][training.pretrain_engine] Budget: $400 | Cost rate: $13.16/hr [2026-10-19 18:40:50][INFO][training.pretrain_engine] Max training hours: 30.4h [2026-10-19 18:40:50][INFO][training.pretrain_engine] World size: 4 [2026-10-19 18:40:50][INFO][training.pretrain_engine] ====================================================================== [2026-10-19 18:40:50][INFO][training.pretrain_engine] ================================================== [2026-10-19 18:40:50][INFO][training.pretrain_engine] PHASE 1: SKIPPED (Starting directly on Phase 2: Core Pretraining @ seq=2048) [2026-10-19 18:40:50][INFO][training.pretrain_engine] ARG local window updated to 64 [2026-10-19 18:40:50][INFO][training.pretrain_engine] ================================================== [2026-10-19 18:40:50][INFO][training.pretrain_engine] PHASE 2: Core Pre-Training (15.0B tokens, seq=2048) [2026-10-19 18:40:50][INFO][training.pretrain_engine] 🚀 Wrapped model in DDP (world_size=4, static_graph=False, find_unused_parameters=True) [2026-10-19 18:40:50][INFO][training.pretrain_engine] ====================================================================== [2026-10-19 18:40:50][INFO][training.pretrain_engine] FORGE-500M PRETRAINING STARTED (plain PyTorch DDP, no DeepSpeed) [2026-10-19 18:40:50][INFO][training.pretrain_engine] Target: 15B tokens (Chinchilla-optimal for 500M) [2026-10-19 18:40:50][INFO][training.pretrain_engine] Budget: $400 | Cost rate: $13.16/hr [2026-10-19 18:40:50][INFO][training.pretrain_engine] Max training hours: 30.4h [2026-10-19 18:40:50][INFO][training.pretrain_engine] World size: 4 [2026-10-19 18:40:50][INFO][training.pretrain_engine] ====================================================================== [2026-10-19 18:40:50][INFO][training.pretrain_engine] ================================================== [2026-10-19 18:40:50][INFO][training.pretrain_engine] PHASE 1: SKIPPED (Starting directly on Phase 2: Core Pretraining @ seq=2048) [2026-10-19 18:40:50][INFO][training.pretrain_engine] ARG local window updated to 64 [2026-10-19 18:40:50][INFO][training.pretrain_engine] ================================================== [2026-10-19 18:40:50][INFO][training.pretrain_engine] PHASE 2: Core Pre-Training (15.0B tokens, seq=2048) [2026-10-19 18:40:50][INFO][training.pretrain_engine] 🚀 Wrapped model in DDP (world_size=4, static_graph=False, find_unused_parameters=True) [2026-10-19 18:40:50][INFO][training.pretrain_engine] ====================================================================== [2026-10-19 18:40:50][INFO][training.pretrain_engine] FORGE-500M PRETRAINING STARTED (plain PyTorch DDP, no DeepSpeed) [2026-10-19 18:40:50][INFO][training.pretrain_engine] Target: 15B tokens (Chinchilla-optimal for 500M) [2026-10-19 18:40:50][INFO][training.pretrain_engine] Budget: $400 | Cost rate: $13.16/hr [2026-10-19 18:40:50][INFO][training.pretrain_engine] Max training hours: 30.4h [2026-10-19 18:40:50][INFO][training.pretrain_engine] World size: 4 [2026-10-19 18:40:50][INFO][training.pretrain_engine] ====================================================================== [2026-10-19 18:40:50][INFO][training.pretrain_engine] ================================================== [2026-10-19 18:40:50][INFO][training.pretrain_engine] PHASE 1: SKIPPED (Starting directly on Phase 2: Core Pretraining @ seq=2048) [2026-10-19 18:40:50][INFO][training.pretrain_engine] ARG local window updated to 64 [2026-10-19 18:40:50][INFO][training.pretrain_engine] ================================================== [2026-10-19 18:40:50][INFO][training.pretrain_engine] PHASE 2: Core Pre-Training (15.0B tokens, seq=2048) [2026-10-19 18:40:51][INFO][training.pretrain_engine] Starting global step 1 (accumulating 32 micro-batches)... [2026-10-19 18:40:51][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 18:41:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.9035 [2026-10-19 18:41:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.9112 [2026-10-19 18:41:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.7577 [2026-10-19 18:41:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.9862 [2026-10-19 18:41:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.9760 [2026-10-19 18:41:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.9477 [2026-10-19 18:41:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.9820 [2026-10-19 18:41:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.9374 [2026-10-19 18:41:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.9236 [2026-10-19 18:41:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.7824 [2026-10-19 18:41:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.6386 [2026-10-19 18:41:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.8434 [2026-10-19 18:42:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.9118 [2026-10-19 18:42:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.9119 [2026-10-19 18:42:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.8903 [2026-10-19 18:42:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.8732 [2026-10-19 18:42:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=12.0618 [2026-10-19 18:42:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=12.2140 [2026-10-19 18:42:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=12.0083 [2026-10-19 18:42:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=12.0583 [2026-10-19 18:42:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.8306 [2026-10-19 18:42:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.8732 [2026-10-19 18:42:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.8343 [2026-10-19 18:42:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.8600 [2026-10-19 18:43:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.9065 [2026-10-19 18:43:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.8941 [2026-10-19 18:43:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.8560 [2026-10-19 18:43:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.8413 [2026-10-19 18:43:15][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,262 | MFU: 1.7% [2026-10-19 18:43:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.9489 [2026-10-19 18:43:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.8528 [2026-10-19 18:43:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.8717 [2026-10-19 18:43:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.7887 [2026-10-19 18:43:20][INFO][training.pretrain_engine] [Ph2|Step 1] loss=11.9209 | ppl=150381.77 | aux=0.02775 | lr=3.45e-05 | gnorm=1.844 | tok/s=28,053 | MFU=6.7% | step=149516ms | tokens=0.008B | mem=4.2/32.7GB | cost=$0.55/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 23:10 (148.4h) | Phase3: SKIPPED | SFT: Oct 26 04:07 (153.4h) | DPO (Final): Oct 26 04:42 (154.0h) [2026-10-19 18:43:20][INFO][training.pretrain_engine] Starting global step 2 (accumulating 32 micro-batches)... [2026-10-19 18:43:20][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 18:43:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.9361 [2026-10-19 18:43:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.8887 [2026-10-19 18:43:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.8995 [2026-10-19 18:43:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=12.0114 [2026-10-19 18:43:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.7491 [2026-10-19 18:43:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.7821 [2026-10-19 18:43:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.7966 [2026-10-19 18:43:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.7879 [2026-10-19 18:44:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.8301 [2026-10-19 18:44:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.9862 [2026-10-19 18:44:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=12.0104 [2026-10-19 18:44:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.9846 [2026-10-19 18:44:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.7746 [2026-10-19 18:44:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.7944 [2026-10-19 18:44:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.8869 [2026-10-19 18:44:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.9097 [2026-10-19 18:44:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.8845 [2026-10-19 18:44:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=12.0112 [2026-10-19 18:44:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.8995 [2026-10-19 18:44:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.9744 [2026-10-19 18:45:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.6216 [2026-10-19 18:45:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.6870 [2026-10-19 18:45:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.7661 [2026-10-19 18:45:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.6583 [2026-10-19 18:45:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.9096 [2026-10-19 18:45:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.9309 [2026-10-19 18:45:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.8575 [2026-10-19 18:45:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.9397 [2026-10-19 18:45:39][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,588 | MFU: 1.8% [2026-10-19 18:45:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.8209 [2026-10-19 18:45:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.7251 [2026-10-19 18:45:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.8915 [2026-10-19 18:45:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.8352 [2026-10-19 18:45:43][INFO][training.pretrain_engine] [Ph2|Step 2] loss=11.8561 | ppl=140936.02 | aux=0.02830 | lr=3.51e-05 | gnorm=1.914 | tok/s=28,683 | MFU=6.9% | step=146229ms | tokens=0.009B | mem=4.2/32.7GB | cost=$1.07/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 19:56 (145.2h) | Phase3: SKIPPED | SFT: Oct 26 00:46 (150.0h) | DPO (Final): Oct 26 01:21 (150.6h) [2026-10-19 18:45:43][INFO][training.pretrain_engine] Starting global step 3 (accumulating 32 micro-batches)... [2026-10-19 18:45:43][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 18:46:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.7500 [2026-10-19 18:46:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.7409 [2026-10-19 18:46:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.7487 [2026-10-19 18:46:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.6577 [2026-10-19 18:46:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.7833 [2026-10-19 18:46:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.8605 [2026-10-19 18:46:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=12.0302 [2026-10-19 18:46:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.9801 [2026-10-19 18:46:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.9547 [2026-10-19 18:46:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.3147 [2026-10-19 18:46:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.8276 [2026-10-19 18:46:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.8551 [2026-10-19 18:46:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.8103 [2026-10-19 18:46:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.7674 [2026-10-19 18:46:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.8320 [2026-10-19 18:46:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.8306 [2026-10-19 18:47:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.8208 [2026-10-19 18:47:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.6114 [2026-10-19 18:47:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.6797 [2026-10-19 18:47:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.6720 [2026-10-19 18:47:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.6626 [2026-10-19 18:47:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.7271 [2026-10-19 18:47:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.8221 [2026-10-19 18:47:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.7792 [2026-10-19 18:47:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.5677 [2026-10-19 18:47:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.8981 [2026-10-19 18:47:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.7693 [2026-10-19 18:47:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.6974 [2026-10-19 18:48:02][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,563 | MFU: 1.8% [2026-10-19 18:48:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.7437 [2026-10-19 18:48:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.7618 [2026-10-19 18:48:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.6562 [2026-10-19 18:48:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.7878 [2026-10-19 18:48:06][INFO][training.pretrain_engine] [Ph2|Step 3] loss=11.8168 | ppl=135510.48 | aux=0.02857 | lr=3.57e-05 | gnorm=2.000 | tok/s=28,886 | MFU=6.9% | step=145201ms | tokens=0.010B | mem=4.2/32.7GB | cost=$1.60/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 18:56 (144.1h) | Phase3: SKIPPED | SFT: Oct 25 23:45 (149.0h) | DPO (Final): Oct 26 00:19 (149.5h) [2026-10-19 18:48:06][INFO][training.pretrain_engine] Starting global step 4 (accumulating 32 micro-batches)... [2026-10-19 18:48:06][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 18:48:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=12.0461 [2026-10-19 18:48:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.9629 [2026-10-19 18:48:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.8080 [2026-10-19 18:48:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.9610 [2026-10-19 18:48:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.9204 [2026-10-19 18:48:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.9318 [2026-10-19 18:48:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.7124 [2026-10-19 18:48:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.9149 [2026-10-19 18:49:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.7053 [2026-10-19 18:49:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.7213 [2026-10-19 18:49:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.6400 [2026-10-19 18:49:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.8039 [2026-10-19 18:49:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.7443 [2026-10-19 18:49:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.6947 [2026-10-19 18:49:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.7814 [2026-10-19 18:49:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.7142 [2026-10-19 18:49:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.4536 [2026-10-19 18:49:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.5977 [2026-10-19 18:49:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.5857 [2026-10-19 18:49:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.6255 [2026-10-19 18:49:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.7533 [2026-10-19 18:49:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.7772 [2026-10-19 18:49:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.7088 [2026-10-19 18:49:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.7187 [2026-10-19 18:50:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.7033 [2026-10-19 18:50:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.9814 [2026-10-19 18:50:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.8483 [2026-10-19 18:50:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.8996 [2026-10-19 18:50:25][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,568 | MFU: 1.8% [2026-10-19 18:50:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.6399 [2026-10-19 18:50:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.6737 [2026-10-19 18:50:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.5873 [2026-10-19 18:50:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.6342 [2026-10-19 18:50:30][INFO][training.pretrain_engine] [Ph2|Step 4] loss=11.8188 | ppl=135780.98 | aux=0.02846 | lr=3.62e-05 | gnorm=1.773 | tok/s=28,988 | MFU=7.0% | step=144689ms | tokens=0.012B | mem=4.2/32.7GB | cost=$2.12/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 18:27 (143.6h) | Phase3: SKIPPED | SFT: Oct 25 23:15 (148.4h) | DPO (Final): Oct 25 23:49 (149.0h) [2026-10-19 18:50:30][INFO][training.pretrain_engine] Starting global step 5 (accumulating 32 micro-batches)... [2026-10-19 18:50:30][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 18:50:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.5503 [2026-10-19 18:50:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.8056 [2026-10-19 18:50:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.5426 [2026-10-19 18:50:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.5718 [2026-10-19 18:51:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.8193 [2026-10-19 18:51:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.8068 [2026-10-19 18:51:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.6973 [2026-10-19 18:51:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.7912 [2026-10-19 18:51:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.5218 [2026-10-19 18:51:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.4893 [2026-10-19 18:51:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.5222 [2026-10-19 18:51:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.4774 [2026-10-19 18:51:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.6307 [2026-10-19 18:51:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.6821 [2026-10-19 18:51:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.8071 [2026-10-19 18:51:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.7158 [2026-10-19 18:51:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.8155 [2026-10-19 18:51:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.8291 [2026-10-19 18:51:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.9485 [2026-10-19 18:51:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.8366 [2026-10-19 18:52:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.6678 [2026-10-19 18:52:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.6258 [2026-10-19 18:52:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.5807 [2026-10-19 18:52:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.5760 [2026-10-19 18:52:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.8952 [2026-10-19 18:52:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.8912 [2026-10-19 18:52:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.8359 [2026-10-19 18:52:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.8388 [2026-10-19 18:52:48][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,597 | MFU: 1.8% [2026-10-19 18:52:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.8197 [2026-10-19 18:52:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.6467 [2026-10-19 18:52:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.8201 [2026-10-19 18:52:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.7651 [2026-10-19 18:52:52][INFO][training.pretrain_engine] [Ph2|Step 5] loss=11.6802 | ppl=118202.26 | aux=0.02923 | lr=3.68e-05 | gnorm=2.016 | tok/s=29,067 | MFU=7.0% | step=144298ms | tokens=0.013B | mem=4.2/32.7GB | cost=$2.64/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 18:06 (143.2h) | Phase3: SKIPPED | SFT: Oct 25 22:53 (148.0h) | DPO (Final): Oct 25 23:27 (148.6h) [2026-10-19 18:52:52][INFO][training.pretrain_engine] Starting global step 6 (accumulating 32 micro-batches)... [2026-10-19 18:52:52][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 18:53:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.6590 [2026-10-19 18:53:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.7818 [2026-10-19 18:53:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.8373 [2026-10-19 18:53:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.7349 [2026-10-19 18:53:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.7382 [2026-10-19 18:53:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.7337 [2026-10-19 18:53:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.8965 [2026-10-19 18:53:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.8629 [2026-10-19 18:53:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.4330 [2026-10-19 18:53:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.4717 [2026-10-19 18:53:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.4697 [2026-10-19 18:53:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.3405 [2026-10-19 18:54:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.7921 [2026-10-19 18:54:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.7595 [2026-10-19 18:54:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.8688 [2026-10-19 18:54:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.6310 [2026-10-19 18:54:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.7987 [2026-10-19 18:54:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.5900 [2026-10-19 18:54:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.5866 [2026-10-19 18:54:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.7934 [2026-10-19 18:54:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.3802 [2026-10-19 18:54:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.4370 [2026-10-19 18:54:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.4379 [2026-10-19 18:54:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.3796 [2026-10-19 18:54:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.5406 [2026-10-19 18:54:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.5886 [2026-10-19 18:54:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.4904 [2026-10-19 18:54:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.5427 [2026-10-19 18:55:11][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,582 | MFU: 1.8% [2026-10-19 18:55:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.3774 [2026-10-19 18:55:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.4677 [2026-10-19 18:55:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.5307 [2026-10-19 18:55:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.3811 [2026-10-19 18:55:15][INFO][training.pretrain_engine] [Ph2|Step 6] loss=11.6566 | ppl=115452.86 | aux=0.02940 | lr=3.74e-05 | gnorm=1.969 | tok/s=29,115 | MFU=7.0% | step=144062ms | tokens=0.014B | mem=4.2/32.7GB | cost=$3.16/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 17:54 (143.0h) | Phase3: SKIPPED | SFT: Oct 25 22:40 (147.8h) | DPO (Final): Oct 25 23:14 (148.3h) [2026-10-19 18:55:15][INFO][training.pretrain_engine] Starting global step 7 (accumulating 32 micro-batches)... [2026-10-19 18:55:15][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 18:55:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.4550 [2026-10-19 18:55:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.5517 [2026-10-19 18:55:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.5504 [2026-10-19 18:55:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.4494 [2026-10-19 18:55:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.4982 [2026-10-19 18:55:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.4417 [2026-10-19 18:55:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.5932 [2026-10-19 18:55:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.5439 [2026-10-19 18:56:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.4258 [2026-10-19 18:56:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.5610 [2026-10-19 18:56:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.5012 [2026-10-19 18:56:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.3304 [2026-10-19 18:56:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.6984 [2026-10-19 18:56:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.7073 [2026-10-19 18:56:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.8062 [2026-10-19 18:56:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.9138 [2026-10-19 18:56:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.2527 [2026-10-19 18:56:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.3590 [2026-10-19 18:56:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.5088 [2026-10-19 18:56:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.2480 [2026-10-19 18:57:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.3778 [2026-10-19 18:57:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.4036 [2026-10-19 18:57:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.2926 [2026-10-19 18:57:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.3090 [2026-10-19 18:57:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.4069 [2026-10-19 18:57:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.5873 [2026-10-19 18:57:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.5733 [2026-10-19 18:57:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.6808 [2026-10-19 18:57:34][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,574 | MFU: 1.8% [2026-10-19 18:57:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.3980 [2026-10-19 18:57:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.4576 [2026-10-19 18:57:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.3633 [2026-10-19 18:57:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.3992 [2026-10-19 18:57:38][INFO][training.pretrain_engine] [Ph2|Step 7] loss=11.5686 | ppl=105727.97 | aux=0.02995 | lr=3.79e-05 | gnorm=2.125 | tok/s=29,147 | MFU=7.0% | step=143901ms | tokens=0.015B | mem=4.2/32.7GB | cost=$3.69/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 17:46 (142.8h) | Phase3: SKIPPED | SFT: Oct 25 22:32 (147.6h) | DPO (Final): Oct 25 23:06 (148.1h) [2026-10-19 18:57:38][INFO][training.pretrain_engine] Starting global step 8 (accumulating 32 micro-batches)... [2026-10-19 18:57:38][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 18:57:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.6500 [2026-10-19 18:57:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.5165 [2026-10-19 18:57:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.8268 [2026-10-19 18:57:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.6581 [2026-10-19 18:58:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.5963 [2026-10-19 18:58:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.6030 [2026-10-19 18:58:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.4247 [2026-10-19 18:58:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.3045 [2026-10-19 18:58:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.2795 [2026-10-19 18:58:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.3048 [2026-10-19 18:58:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.3497 [2026-10-19 18:58:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.2452 [2026-10-19 18:58:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.9711 [2026-10-19 18:58:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.7198 [2026-10-19 18:58:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.8859 [2026-10-19 18:58:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.8707 [2026-10-19 18:59:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.5671 [2026-10-19 18:59:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.6528 [2026-10-19 18:59:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.5050 [2026-10-19 18:59:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.5383 [2026-10-19 18:59:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.6379 [2026-10-19 18:59:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.6733 [2026-10-19 18:59:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.7178 [2026-10-19 18:59:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.7135 [2026-10-19 18:59:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.6311 [2026-10-19 18:59:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.5697 [2026-10-19 18:59:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.4616 [2026-10-19 18:59:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.5619 [2026-10-19 18:59:57][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,572 | MFU: 1.8% [2026-10-19 19:00:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.5146 [2026-10-19 19:00:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.4719 [2026-10-19 19:00:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.2225 [2026-10-19 19:00:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.2432 [2026-10-19 19:00:01][INFO][training.pretrain_engine] [Ph2|Step 8] loss=11.5585 | ppl=104662.55 | aux=0.02982 | lr=3.85e-05 | gnorm=1.992 | tok/s=29,170 | MFU=7.0% | step=143790ms | tokens=0.016B | mem=4.2/32.7GB | cost=$4.21/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 17:41 (142.7h) | Phase3: SKIPPED | SFT: Oct 25 22:27 (147.5h) | DPO (Final): Oct 25 23:01 (148.0h) [2026-10-19 19:00:01][INFO][training.pretrain_engine] Starting global step 9 (accumulating 32 micro-batches)... [2026-10-19 19:00:01][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 19:00:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.7444 [2026-10-19 19:00:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.5564 [2026-10-19 19:00:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.6689 [2026-10-19 19:00:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.5926 [2026-10-19 19:00:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.6620 [2026-10-19 19:00:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.6564 [2026-10-19 19:00:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.6467 [2026-10-19 19:00:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.6948 [2026-10-19 19:00:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.4846 [2026-10-19 19:00:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.7477 [2026-10-19 19:00:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.7648 [2026-10-19 19:00:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.7102 [2026-10-19 19:01:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.1295 [2026-10-19 19:01:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.2654 [2026-10-19 19:01:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.2302 [2026-10-19 19:01:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.2587 [2026-10-19 19:01:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.8955 [2026-10-19 19:01:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.4725 [2026-10-19 19:01:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.6037 [2026-10-19 19:01:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.6709 [2026-10-19 19:01:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.8904 [2026-10-19 19:01:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.8926 [2026-10-19 19:01:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=12.0026 [2026-10-19 19:01:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.8805 [2026-10-19 19:02:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.4852 [2026-10-19 19:02:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.6859 [2026-10-19 19:02:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.4433 [2026-10-19 19:02:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.4353 [2026-10-19 19:02:19][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,584 | MFU: 1.8% [2026-10-19 19:02:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.5130 [2026-10-19 19:02:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.4318 [2026-10-19 19:02:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.4520 [2026-10-19 19:02:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.4995 [2026-10-19 19:02:24][INFO][training.pretrain_engine] [Ph2|Step 9] loss=11.4902 | ppl=97755.70 | aux=0.03010 | lr=3.91e-05 | gnorm=1.953 | tok/s=29,189 | MFU=7.0% | step=143695ms | tokens=0.017B | mem=4.2/32.7GB | cost=$4.73/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 17:37 (142.6h) | Phase3: SKIPPED | SFT: Oct 25 22:23 (147.3h) | DPO (Final): Oct 25 22:57 (147.9h) [2026-10-19 19:02:27][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step9 | Tokens: 0.017B | Spent: $4.74 | Remaining: $395.26 [2026-10-19 19:02:27][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step9 [2026-10-19 19:02:27][INFO][training.pretrain_engine] Starting global step 10 (accumulating 32 micro-batches)... [2026-10-19 19:02:27][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 19:02:27][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-19 19:02:27][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-19 19:02:27][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-19 19:02:27][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step9 to Phase-Technologies/forge-500m-pretrain (path: phase2_step9)... [2026-10-19 19:02:27][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-19 19:02:41][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-19 19:02:41][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step9 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-19 19:02:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.4943 [2026-10-19 19:02:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.4575 [2026-10-19 19:02:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.3066 [2026-10-19 19:02:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.1795 [2026-10-19 19:03:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.3767 [2026-10-19 19:03:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.3516 [2026-10-19 19:03:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.3254 [2026-10-19 19:03:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.2086 [2026-10-19 19:03:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.6213 [2026-10-19 19:03:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.6607 [2026-10-19 19:03:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.6325 [2026-10-19 19:03:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.5718 [2026-10-19 19:03:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.3930 [2026-10-19 19:03:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.3824 [2026-10-19 19:03:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.4888 [2026-10-19 19:03:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.4974 [2026-10-19 19:03:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.4695 [2026-10-19 19:03:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.1799 [2026-10-19 19:03:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.6636 [2026-10-19 19:03:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.6628 [2026-10-19 19:04:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.1254 [2026-10-19 19:04:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.1925 [2026-10-19 19:04:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.3916 [2026-10-19 19:04:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.5338 [2026-10-19 19:04:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.7648 [2026-10-19 19:04:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.6038 [2026-10-19 19:04:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.7116 [2026-10-19 19:04:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.7091 [2026-10-19 19:04:45][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,579 | MFU: 1.8% [2026-10-19 19:04:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.4478 [2026-10-19 19:04:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.4691 [2026-10-19 19:04:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.5218 [2026-10-19 19:04:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.5751 [2026-10-19 19:04:50][INFO][training.pretrain_engine] [Ph2|Step 10] loss=11.5175 | ppl=100462.44 | aux=0.02993 | lr=3.96e-05 | gnorm=1.898 | tok/s=29,207 | MFU=7.0% | step=143608ms | tokens=0.018B | mem=4.2/32.7GB | cost=$5.27/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 17:34 (142.5h) | Phase3: SKIPPED | SFT: Oct 25 22:19 (147.2h) | DPO (Final): Oct 25 22:53 (147.8h) [2026-10-19 19:04:50][INFO][training.pretrain_engine] Starting global step 11 (accumulating 32 micro-batches)... [2026-10-19 19:04:50][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 19:05:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.4213 [2026-10-19 19:05:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.4757 [2026-10-19 19:05:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.5833 [2026-10-19 19:05:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.2191 [2026-10-19 19:05:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.7134 [2026-10-19 19:05:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.5522 [2026-10-19 19:05:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.7188 [2026-10-19 19:05:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.7275 [2026-10-19 19:05:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.7659 [2026-10-19 19:05:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.8041 [2026-10-19 19:05:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.7221 [2026-10-19 19:05:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.7032 [2026-10-19 19:06:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.2048 [2026-10-19 19:06:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.1685 [2026-10-19 19:06:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.2554 [2026-10-19 19:06:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.0643 [2026-10-19 19:06:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.2704 [2026-10-19 19:06:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.5137 [2026-10-19 19:06:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.3796 [2026-10-19 19:06:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.6109 [2026-10-19 19:06:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.6779 [2026-10-19 19:06:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.6828 [2026-10-19 19:06:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.6966 [2026-10-19 19:06:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.8058 [2026-10-19 19:06:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.7736 [2026-10-19 19:06:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.7107 [2026-10-19 19:06:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.7248 [2026-10-19 19:06:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.9246 [2026-10-19 19:07:08][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,603 | MFU: 1.8% [2026-10-19 19:07:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.4963 [2026-10-19 19:07:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.3124 [2026-10-19 19:07:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.2157 [2026-10-19 19:07:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.4901 [2026-10-19 19:07:13][INFO][training.pretrain_engine] [Ph2|Step 11] loss=11.4387 | ppl=92845.27 | aux=0.03033 | lr=4.02e-05 | gnorm=1.969 | tok/s=29,213 | MFU=7.0% | step=143575ms | tokens=0.019B | mem=4.2/32.7GB | cost=$5.79/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 17:34 (142.4h) | Phase3: SKIPPED | SFT: Oct 25 22:19 (147.2h) | DPO (Final): Oct 25 22:53 (147.8h) [2026-10-19 19:07:13][INFO][training.pretrain_engine] Starting global step 12 (accumulating 32 micro-batches)... [2026-10-19 19:07:13][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 19:07:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.0741 [2026-10-19 19:07:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.1791 [2026-10-19 19:07:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.3732 [2026-10-19 19:07:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.3742 [2026-10-19 19:07:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.4276 [2026-10-19 19:07:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.4844 [2026-10-19 19:07:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.5520 [2026-10-19 19:07:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.8592 [2026-10-19 19:08:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.4205 [2026-10-19 19:08:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.2712 [2026-10-19 19:08:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.3365 [2026-10-19 19:08:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.4277 [2026-10-19 19:08:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.0770 [2026-10-19 19:08:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.0781 [2026-10-19 19:08:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.1694 [2026-10-19 19:08:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.0679 [2026-10-19 19:08:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.6398 [2026-10-19 19:08:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.7354 [2026-10-19 19:08:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.6556 [2026-10-19 19:08:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.7815 [2026-10-19 19:09:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.4061 [2026-10-19 19:09:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.4116 [2026-10-19 19:09:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.4146 [2026-10-19 19:09:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.4481 [2026-10-19 19:09:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.2684 [2026-10-19 19:09:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.4681 [2026-10-19 19:09:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.3284 [2026-10-19 19:09:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.8932 [2026-10-19 19:09:31][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,606 | MFU: 1.8% [2026-10-19 19:09:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.5552 [2026-10-19 19:09:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.4524 [2026-10-19 19:09:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.3695 [2026-10-19 19:09:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.3797 [2026-10-19 19:09:36][INFO][training.pretrain_engine] [Ph2|Step 12] loss=11.3747 | ppl=87088.37 | aux=0.03068 | lr=4.08e-05 | gnorm=1.945 | tok/s=29,223 | MFU=7.0% | step=143528ms | tokens=0.020B | mem=4.2/32.7GB | cost=$6.31/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 17:33 (142.4h) | Phase3: SKIPPED | SFT: Oct 25 22:18 (147.1h) | DPO (Final): Oct 25 22:52 (147.7h) [2026-10-19 19:09:36][INFO][training.pretrain_engine] Starting global step 13 (accumulating 32 micro-batches)... [2026-10-19 19:09:36][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 19:09:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.2934 [2026-10-19 19:09:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.4503 [2026-10-19 19:09:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.4539 [2026-10-19 19:09:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.6078 [2026-10-19 19:10:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.3969 [2026-10-19 19:10:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.9520 [2026-10-19 19:10:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.2116 [2026-10-19 19:10:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.1031 [2026-10-19 19:10:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.4331 [2026-10-19 19:10:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.5630 [2026-10-19 19:10:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.5756 [2026-10-19 19:10:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.2976 [2026-10-19 19:10:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.0261 [2026-10-19 19:10:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.1690 [2026-10-19 19:10:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.2131 [2026-10-19 19:10:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.1413 [2026-10-19 19:11:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.3750 [2026-10-19 19:11:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.3560 [2026-10-19 19:11:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.2231 [2026-10-19 19:11:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.4419 [2026-10-19 19:11:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.5188 [2026-10-19 19:11:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.3335 [2026-10-19 19:11:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.3902 [2026-10-19 19:11:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.4638 [2026-10-19 19:11:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.2068 [2026-10-19 19:11:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.2713 [2026-10-19 19:11:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.5567 [2026-10-19 19:11:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.2434 [2026-10-19 19:11:55][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,566 | MFU: 1.8% [2026-10-19 19:12:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.3925 [2026-10-19 19:12:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.4272 [2026-10-19 19:12:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.2888 [2026-10-19 19:12:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.4275 [2026-10-19 19:12:00][INFO][training.pretrain_engine] [Ph2|Step 13] loss=11.3315 | ppl=83404.42 | aux=0.03067 | lr=4.13e-05 | gnorm=1.922 | tok/s=29,223 | MFU=7.0% | step=143526ms | tokens=0.021B | mem=4.2/32.7GB | cost=$6.84/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 17:34 (142.4h) | Phase3: SKIPPED | SFT: Oct 25 22:20 (147.1h) | DPO (Final): Oct 25 22:54 (147.7h) [2026-10-19 19:12:00][INFO][training.pretrain_engine] Starting global step 14 (accumulating 32 micro-batches)... [2026-10-19 19:12:00][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 19:12:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.5304 [2026-10-19 19:12:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.3219 [2026-10-19 19:12:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.5248 [2026-10-19 19:12:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.3074 [2026-10-19 19:12:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.4602 [2026-10-19 19:12:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.6199 [2026-10-19 19:12:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.6587 [2026-10-19 19:12:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.5396 [2026-10-19 19:12:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.0141 [2026-10-19 19:12:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.0128 [2026-10-19 19:12:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.0936 [2026-10-19 19:12:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.1031 [2026-10-19 19:13:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.4668 [2026-10-19 19:13:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.5373 [2026-10-19 19:13:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.5383 [2026-10-19 19:13:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.5149 [2026-10-19 19:13:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.3055 [2026-10-19 19:13:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.2686 [2026-10-19 19:13:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.3046 [2026-10-19 19:13:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.3209 [2026-10-19 19:13:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=12.0901 [2026-10-19 19:13:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=12.1199 [2026-10-19 19:13:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=12.1521 [2026-10-19 19:13:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.9003 [2026-10-19 19:14:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.2235 [2026-10-19 19:14:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.3649 [2026-10-19 19:14:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.2590 [2026-10-19 19:14:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.2723 [2026-10-19 19:14:18][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,589 | MFU: 1.8% [2026-10-19 19:14:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.2447 [2026-10-19 19:14:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.3735 [2026-10-19 19:14:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.0045 [2026-10-19 19:14:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.0988 [2026-10-19 19:14:23][INFO][training.pretrain_engine] [Ph2|Step 14] loss=11.3461 | ppl=84635.77 | aux=0.03060 | lr=4.19e-05 | gnorm=1.812 | tok/s=29,228 | MFU=7.0% | step=143505ms | tokens=0.022B | mem=4.2/32.7GB | cost=$7.36/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 17:35 (142.4h) | Phase3: SKIPPED | SFT: Oct 25 22:20 (147.1h) | DPO (Final): Oct 25 22:54 (147.7h) [2026-10-19 19:14:23][INFO][training.pretrain_engine] Starting global step 15 (accumulating 32 micro-batches)... [2026-10-19 19:14:23][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 19:14:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.5634 [2026-10-19 19:14:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.2556 [2026-10-19 19:14:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.2271 [2026-10-19 19:14:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.2084 [2026-10-19 19:14:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.5463 [2026-10-19 19:14:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.4980 [2026-10-19 19:14:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.4633 [2026-10-19 19:14:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.5459 [2026-10-19 19:15:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.3570 [2026-10-19 19:15:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.3312 [2026-10-19 19:15:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.4386 [2026-10-19 19:15:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.5681 [2026-10-19 19:15:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.0152 [2026-10-19 19:15:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.3482 [2026-10-19 19:15:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.1770 [2026-10-19 19:15:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.2727 [2026-10-19 19:15:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.9950 [2026-10-19 19:15:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.0457 [2026-10-19 19:15:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.9648 [2026-10-19 19:15:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.9705 [2026-10-19 19:16:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.2316 [2026-10-19 19:16:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.4791 [2026-10-19 19:16:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.2843 [2026-10-19 19:16:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.0791 [2026-10-19 19:16:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.4253 [2026-10-19 19:16:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.2040 [2026-10-19 19:16:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.3137 [2026-10-19 19:16:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.3056 [2026-10-19 19:16:41][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,589 | MFU: 1.8% [2026-10-19 19:16:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.2692 [2026-10-19 19:16:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.7074 [2026-10-19 19:16:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.4235 [2026-10-19 19:16:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.6960 [2026-10-19 19:16:46][INFO][training.pretrain_engine] [Ph2|Step 15] loss=11.2574 | ppl=77448.11 | aux=0.03096 | lr=4.25e-05 | gnorm=1.883 | tok/s=29,232 | MFU=7.0% | step=143484ms | tokens=0.023B | mem=4.2/32.7GB | cost=$7.88/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 17:35 (142.3h) | Phase3: SKIPPED | SFT: Oct 25 22:21 (147.1h) | DPO (Final): Oct 25 22:55 (147.6h) [2026-10-19 19:16:46][INFO][training.pretrain_engine] Starting global step 16 (accumulating 32 micro-batches)... [2026-10-19 19:16:46][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 19:17:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.1581 [2026-10-19 19:17:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.2820 [2026-10-19 19:17:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.2748 [2026-10-19 19:17:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.1928 [2026-10-19 19:17:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.4052 [2026-10-19 19:17:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.6896 [2026-10-19 19:17:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.5966 [2026-10-19 19:17:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.5745 [2026-10-19 19:17:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.3919 [2026-10-19 19:17:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.2559 [2026-10-19 19:17:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.4503 [2026-10-19 19:17:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.3812 [2026-10-19 19:17:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.6947 [2026-10-19 19:17:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.7219 [2026-10-19 19:17:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.6620 [2026-10-19 19:17:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.6500 [2026-10-19 19:18:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.6680 [2026-10-19 19:18:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.3120 [2026-10-19 19:18:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.4192 [2026-10-19 19:18:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.4566 [2026-10-19 19:18:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.4901 [2026-10-19 19:18:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.4190 [2026-10-19 19:18:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.4143 [2026-10-19 19:18:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.3676 [2026-10-19 19:18:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.1429 [2026-10-19 19:18:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.2874 [2026-10-19 19:18:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.2798 [2026-10-19 19:18:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.3562 [2026-10-19 19:19:04][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,595 | MFU: 1.8% [2026-10-19 19:19:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.0755 [2026-10-19 19:19:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.8918 [2026-10-19 19:19:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.1878 [2026-10-19 19:19:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.0202 [2026-10-19 19:19:09][INFO][training.pretrain_engine] [Ph2|Step 16] loss=11.3132 | ppl=81897.80 | aux=0.03054 | lr=4.30e-05 | gnorm=1.742 | tok/s=29,234 | MFU=7.0% | step=143473ms | tokens=0.024B | mem=4.2/32.7GB | cost=$8.41/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 17:37 (142.3h) | Phase3: SKIPPED | SFT: Oct 25 22:22 (147.0h) | DPO (Final): Oct 25 22:56 (147.6h) [2026-10-19 19:19:09][INFO][training.pretrain_engine] Starting global step 17 (accumulating 32 micro-batches)... [2026-10-19 19:19:09][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 19:19:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.5680 [2026-10-19 19:19:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.5670 [2026-10-19 19:19:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.5006 [2026-10-19 19:19:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.4262 [2026-10-19 19:19:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.9934 [2026-10-19 19:19:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.4990 [2026-10-19 19:19:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.1825 [2026-10-19 19:19:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.0830 [2026-10-19 19:20:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.5595 [2026-10-19 19:20:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.4792 [2026-10-19 19:20:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.6665 [2026-10-19 19:20:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.5225 [2026-10-19 19:20:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.6029 [2026-10-19 19:20:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.6452 [2026-10-19 19:20:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.8000 [2026-10-19 19:20:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.6659 [2026-10-19 19:20:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.0022 [2026-10-19 19:20:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.1617 [2026-10-19 19:20:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.1695 [2026-10-19 19:20:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.1588 [2026-10-19 19:20:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.4433 [2026-10-19 19:20:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.4282 [2026-10-19 19:20:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.4168 [2026-10-19 19:20:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.4560 [2026-10-19 19:21:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.3051 [2026-10-19 19:21:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.5321 [2026-10-19 19:21:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.3624 [2026-10-19 19:21:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.5328 [2026-10-19 19:21:28][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,550 | MFU: 1.8% [2026-10-19 19:21:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.0595 [2026-10-19 19:21:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.0591 [2026-10-19 19:21:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.0468 [2026-10-19 19:21:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.1085 [2026-10-19 19:21:33][INFO][training.pretrain_engine] [Ph2|Step 17] loss=11.2665 | ppl=78156.86 | aux=0.03065 | lr=4.36e-05 | gnorm=1.773 | tok/s=29,235 | MFU=7.0% | step=143469ms | tokens=0.025B | mem=4.2/32.7GB | cost=$8.93/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 17:38 (142.3h) | Phase3: SKIPPED | SFT: Oct 25 22:23 (147.0h) | DPO (Final): Oct 25 22:57 (147.6h) [2026-10-19 19:21:33][INFO][training.pretrain_engine] Starting global step 18 (accumulating 32 micro-batches)... [2026-10-19 19:21:33][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 19:21:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.1684 [2026-10-19 19:21:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.0639 [2026-10-19 19:21:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.0277 [2026-10-19 19:21:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.2262 [2026-10-19 19:22:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.2117 [2026-10-19 19:22:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.4914 [2026-10-19 19:22:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.9269 [2026-10-19 19:22:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.2807 [2026-10-19 19:22:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.1603 [2026-10-19 19:22:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.1503 [2026-10-19 19:22:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.1338 [2026-10-19 19:22:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.1976 [2026-10-19 19:22:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.1047 [2026-10-19 19:22:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.0185 [2026-10-19 19:22:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.1453 [2026-10-19 19:22:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.0380 [2026-10-19 19:23:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.1490 [2026-10-19 19:23:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.1965 [2026-10-19 19:23:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.2446 [2026-10-19 19:23:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.1371 [2026-10-19 19:23:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.1468 [2026-10-19 19:23:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.2116 [2026-10-19 19:23:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.2579 [2026-10-19 19:23:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.1629 [2026-10-19 19:23:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.4181 [2026-10-19 19:23:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.4714 [2026-10-19 19:23:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.4906 [2026-10-19 19:23:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.4825 [2026-10-19 19:23:52][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,543 | MFU: 1.8% [2026-10-19 19:23:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.1302 [2026-10-19 19:23:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.0912 [2026-10-19 19:23:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.2090 [2026-10-19 19:23:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.1170 [2026-10-19 19:23:56][INFO][training.pretrain_engine] [Ph2|Step 18] loss=11.2321 | ppl=75518.06 | aux=0.03074 | lr=4.42e-05 | gnorm=1.750 | tok/s=29,234 | MFU=7.0% | step=143472ms | tokens=0.026B | mem=4.2/32.7GB | cost=$9.46/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 17:40 (142.3h) | Phase3: SKIPPED | SFT: Oct 25 22:25 (147.0h) | DPO (Final): Oct 25 22:59 (147.6h) [2026-10-19 19:24:00][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step18 | Tokens: 0.026B | Spent: $9.47 | Remaining: $390.53 [2026-10-19 19:24:00][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step18 [2026-10-19 19:24:00][INFO][training.pretrain_engine] Starting global step 19 (accumulating 32 micro-batches)... [2026-10-19 19:24:00][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 19:24:00][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-19 19:24:00][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-19 19:24:00][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-19 19:24:00][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step18 to Phase-Technologies/forge-500m-pretrain (path: phase2_step18)... [2026-10-19 19:24:00][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-19 19:24:09][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-19 19:24:09][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step18 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-19 19:24:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.6536 [2026-10-19 19:24:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.7288 [2026-10-19 19:24:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.5982 [2026-10-19 19:24:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.7129 [2026-10-19 19:24:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.0175 [2026-10-19 19:24:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.0553 [2026-10-19 19:24:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.0649 [2026-10-19 19:24:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.2914 [2026-10-19 19:24:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.8936 [2026-10-19 19:24:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.6844 [2026-10-19 19:24:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.8030 [2026-10-19 19:24:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.7209 [2026-10-19 19:25:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.5009 [2026-10-19 19:25:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.5779 [2026-10-19 19:25:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.6412 [2026-10-19 19:25:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.3852 [2026-10-19 19:25:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.9537 [2026-10-19 19:25:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.2587 [2026-10-19 19:25:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.9860 [2026-10-19 19:25:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.1896 [2026-10-19 19:25:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.1244 [2026-10-19 19:25:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.0203 [2026-10-19 19:25:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.2171 [2026-10-19 19:25:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.0486 [2026-10-19 19:26:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.8405 [2026-10-19 19:26:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.8353 [2026-10-19 19:26:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.2147 [2026-10-19 19:26:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.1499 [2026-10-19 19:26:18][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,576 | MFU: 1.8% [2026-10-19 19:26:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.9285 [2026-10-19 19:26:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.7229 [2026-10-19 19:26:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.7930 [2026-10-19 19:26:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.9135 [2026-10-19 19:26:23][INFO][training.pretrain_engine] [Ph2|Step 19] loss=11.1645 | ppl=70580.86 | aux=0.03087 | lr=4.47e-05 | gnorm=1.828 | tok/s=29,240 | MFU=7.0% | step=143445ms | tokens=0.027B | mem=4.2/32.7GB | cost=$9.99/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 17:40 (142.2h) | Phase3: SKIPPED | SFT: Oct 25 22:25 (147.0h) | DPO (Final): Oct 25 23:00 (147.6h) [2026-10-19 19:26:23][INFO][training.pretrain_engine] Starting global step 20 (accumulating 32 micro-batches)... [2026-10-19 19:26:23][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 19:26:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.6052 [2026-10-19 19:26:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.6416 [2026-10-19 19:26:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.6768 [2026-10-19 19:26:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.6729 [2026-10-19 19:26:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.0209 [2026-10-19 19:26:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.0258 [2026-10-19 19:26:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.3445 [2026-10-19 19:26:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.1944 [2026-10-19 19:27:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.3598 [2026-10-19 19:27:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.4808 [2026-10-19 19:27:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.3132 [2026-10-19 19:27:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.2977 [2026-10-19 19:27:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.0488 [2026-10-19 19:27:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.0246 [2026-10-19 19:27:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.1793 [2026-10-19 19:27:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.1161 [2026-10-19 19:27:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.5864 [2026-10-19 19:27:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.2210 [2026-10-19 19:27:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.5314 [2026-10-19 19:27:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.3771 [2026-10-19 19:28:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.3095 [2026-10-19 19:28:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.9417 [2026-10-19 19:28:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.9125 [2026-10-19 19:28:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.9142 [2026-10-19 19:28:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.1285 [2026-10-19 19:28:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.1258 [2026-10-19 19:28:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.2248 [2026-10-19 19:28:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.0517 [2026-10-19 19:28:42][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,556 | MFU: 1.8% [2026-10-19 19:28:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.2765 [2026-10-19 19:28:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.1342 [2026-10-19 19:28:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.3772 [2026-10-19 19:28:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.1471 [2026-10-19 19:28:46][INFO][training.pretrain_engine] [Ph2|Step 20] loss=11.2056 | ppl=73539.77 | aux=0.03059 | lr=4.53e-05 | gnorm=1.664 | tok/s=29,241 | MFU=7.0% | step=143437ms | tokens=0.028B | mem=4.2/32.7GB | cost=$10.52/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 17:42 (142.2h) | Phase3: SKIPPED | SFT: Oct 25 22:27 (147.0h) | DPO (Final): Oct 25 23:01 (147.5h) [2026-10-19 19:28:46][INFO][training.pretrain_engine] Starting global step 21 (accumulating 32 micro-batches)... [2026-10-19 19:28:46][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 19:29:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.2210 [2026-10-19 19:29:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.2713 [2026-10-19 19:29:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.0990 [2026-10-19 19:29:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.9448 [2026-10-19 19:29:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.8922 [2026-10-19 19:29:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.7754 [2026-10-19 19:29:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.7166 [2026-10-19 19:29:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.0086 [2026-10-19 19:29:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.0288 [2026-10-19 19:29:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.9627 [2026-10-19 19:29:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.0981 [2026-10-19 19:29:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.0166 [2026-10-19 19:29:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.5930 [2026-10-19 19:29:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.4338 [2026-10-19 19:29:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.3770 [2026-10-19 19:29:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.5483 [2026-10-19 19:30:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.8482 [2026-10-19 19:30:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.8294 [2026-10-19 19:30:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.9210 [2026-10-19 19:30:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.7682 [2026-10-19 19:30:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.1274 [2026-10-19 19:30:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.2048 [2026-10-19 19:30:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.1886 [2026-10-19 19:30:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.2301 [2026-10-19 19:30:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.4891 [2026-10-19 19:30:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.1278 [2026-10-19 19:30:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.2571 [2026-10-19 19:30:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.2200 [2026-10-19 19:31:05][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,543 | MFU: 1.8% [2026-10-19 19:31:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.4537 [2026-10-19 19:31:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.2626 [2026-10-19 19:31:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.2767 [2026-10-19 19:31:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.2606 [2026-10-19 19:31:10][INFO][training.pretrain_engine] [Ph2|Step 21] loss=11.1431 | ppl=69087.27 | aux=0.03069 | lr=4.59e-05 | gnorm=1.727 | tok/s=29,240 | MFU=7.0% | step=143442ms | tokens=0.029B | mem=4.2/32.7GB | cost=$11.04/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 17:44 (142.2h) | Phase3: SKIPPED | SFT: Oct 25 22:29 (147.0h) | DPO (Final): Oct 25 23:03 (147.5h) [2026-10-19 19:31:10][INFO][training.pretrain_engine] Starting global step 22 (accumulating 32 micro-batches)... [2026-10-19 19:31:10][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 19:31:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.2475 [2026-10-19 19:31:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.4414 [2026-10-19 19:31:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.3499 [2026-10-19 19:31:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.2084 [2026-10-19 19:31:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.8481 [2026-10-19 19:31:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.1461 [2026-10-19 19:31:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.9074 [2026-10-19 19:31:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.9716 [2026-10-19 19:32:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.0645 [2026-10-19 19:32:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.0468 [2026-10-19 19:32:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.9514 [2026-10-19 19:32:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.2069 [2026-10-19 19:32:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.0429 [2026-10-19 19:32:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.9647 [2026-10-19 19:32:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.1377 [2026-10-19 19:32:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.0538 [2026-10-19 19:32:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.0184 [2026-10-19 19:32:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.3307 [2026-10-19 19:32:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.0859 [2026-10-19 19:32:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.0278 [2026-10-19 19:32:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.7593 [2026-10-19 19:32:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.6649 [2026-10-19 19:32:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.6357 [2026-10-19 19:32:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.6392 [2026-10-19 19:33:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.8947 [2026-10-19 19:33:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.7836 [2026-10-19 19:33:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.2169 [2026-10-19 19:33:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.7816 [2026-10-19 19:33:28][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,565 | MFU: 1.8% [2026-10-19 19:33:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.0706 [2026-10-19 19:33:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.0588 [2026-10-19 19:33:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.2952 [2026-10-19 19:33:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.9598 [2026-10-19 19:33:33][INFO][training.pretrain_engine] [Ph2|Step 22] loss=11.0150 | ppl=60777.12 | aux=0.03110 | lr=4.64e-05 | gnorm=1.898 | tok/s=29,242 | MFU=7.0% | step=143436ms | tokens=0.030B | mem=4.2/32.7GB | cost=$11.56/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 17:45 (142.2h) | Phase3: SKIPPED | SFT: Oct 25 22:30 (147.0h) | DPO (Final): Oct 25 23:04 (147.5h) [2026-10-19 19:33:33][INFO][training.pretrain_engine] Starting global step 23 (accumulating 32 micro-batches)... [2026-10-19 19:33:33][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 19:33:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.3208 [2026-10-19 19:33:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.2562 [2026-10-19 19:33:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.2476 [2026-10-19 19:33:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.0870 [2026-10-19 19:34:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.7824 [2026-10-19 19:34:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.9455 [2026-10-19 19:34:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.8439 [2026-10-19 19:34:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.8467 [2026-10-19 19:34:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.9492 [2026-10-19 19:34:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.0736 [2026-10-19 19:34:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.2073 [2026-10-19 19:34:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.2238 [2026-10-19 19:34:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.1158 [2026-10-19 19:34:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.2222 [2026-10-19 19:34:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.1257 [2026-10-19 19:34:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.2708 [2026-10-19 19:35:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.0364 [2026-10-19 19:35:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.0730 [2026-10-19 19:35:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.1269 [2026-10-19 19:35:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.0581 [2026-10-19 19:35:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.8383 [2026-10-19 19:35:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.0740 [2026-10-19 19:35:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.7385 [2026-10-19 19:35:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.1024 [2026-10-19 19:35:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.8135 [2026-10-19 19:35:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.9331 [2026-10-19 19:35:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.8254 [2026-10-19 19:35:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.8429 [2026-10-19 19:35:52][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,538 | MFU: 1.8% [2026-10-19 19:35:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.6612 [2026-10-19 19:35:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.5504 [2026-10-19 19:35:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.4422 [2026-10-19 19:35:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.4618 [2026-10-19 19:35:57][INFO][training.pretrain_engine] [Ph2|Step 23] loss=11.0222 | ppl=61217.71 | aux=0.03087 | lr=4.70e-05 | gnorm=1.812 | tok/s=29,240 | MFU=7.0% | step=143444ms | tokens=0.031B | mem=4.2/32.7GB | cost=$12.09/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 17:47 (142.2h) | Phase3: SKIPPED | SFT: Oct 25 22:32 (146.9h) | DPO (Final): Oct 25 23:07 (147.5h) [2026-10-19 19:35:57][INFO][training.pretrain_engine] Starting global step 24 (accumulating 32 micro-batches)... [2026-10-19 19:35:57][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 19:36:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.2720 [2026-10-19 19:36:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.3887 [2026-10-19 19:36:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.2715 [2026-10-19 19:36:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.3269 [2026-10-19 19:36:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.1792 [2026-10-19 19:36:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.6894 [2026-10-19 19:36:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.4486 [2026-10-19 19:36:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.5518 [2026-10-19 19:36:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.7985 [2026-10-19 19:36:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.6940 [2026-10-19 19:36:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.9130 [2026-10-19 19:36:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.9475 [2026-10-19 19:37:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.0566 [2026-10-19 19:37:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.8312 [2026-10-19 19:37:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.9575 [2026-10-19 19:37:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.7534 [2026-10-19 19:37:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.7534 [2026-10-19 19:37:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.0392 [2026-10-19 19:37:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.1900 [2026-10-19 19:37:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.0577 [2026-10-19 19:37:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.6255 [2026-10-19 19:37:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.5190 [2026-10-19 19:37:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.5444 [2026-10-19 19:37:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.5211 [2026-10-19 19:38:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.2342 [2026-10-19 19:38:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.0865 [2026-10-19 19:38:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.5167 [2026-10-19 19:38:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.3171 [2026-10-19 19:38:15][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,564 | MFU: 1.8% [2026-10-19 19:38:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.8150 [2026-10-19 19:38:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.9695 [2026-10-19 19:38:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.8065 [2026-10-19 19:38:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.9190 [2026-10-19 19:38:20][INFO][training.pretrain_engine] [Ph2|Step 24] loss=11.0903 | ppl=65535.45 | aux=0.03064 | lr=4.76e-05 | gnorm=1.688 | tok/s=29,243 | MFU=7.0% | step=143431ms | tokens=0.033B | mem=4.2/32.7GB | cost=$12.61/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 17:48 (142.2h) | Phase3: SKIPPED | SFT: Oct 25 22:33 (146.9h) | DPO (Final): Oct 25 23:08 (147.5h) [2026-10-19 19:38:20][INFO][training.pretrain_engine] Starting global step 25 (accumulating 32 micro-batches)... [2026-10-19 19:38:20][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 19:38:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.9885 [2026-10-19 19:38:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.8021 [2026-10-19 19:38:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.7704 [2026-10-19 19:38:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.0211 [2026-10-19 19:38:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.6863 [2026-10-19 19:38:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.6281 [2026-10-19 19:38:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.7848 [2026-10-19 19:38:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.7581 [2026-10-19 19:39:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.6629 [2026-10-19 19:39:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.7934 [2026-10-19 19:39:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.7512 [2026-10-19 19:39:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.7406 [2026-10-19 19:39:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.3095 [2026-10-19 19:39:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.0878 [2026-10-19 19:39:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.8371 [2026-10-19 19:39:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.9871 [2026-10-19 19:39:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.0542 [2026-10-19 19:39:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.9488 [2026-10-19 19:39:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.9252 [2026-10-19 19:39:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.9838 [2026-10-19 19:40:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.1366 [2026-10-19 19:40:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.0805 [2026-10-19 19:40:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.9625 [2026-10-19 19:40:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.9863 [2026-10-19 19:40:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.1861 [2026-10-19 19:40:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.1412 [2026-10-19 19:40:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.1205 [2026-10-19 19:40:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.2323 [2026-10-19 19:40:39][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,523 | MFU: 1.8% [2026-10-19 19:40:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.6408 [2026-10-19 19:40:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.7450 [2026-10-19 19:40:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.7956 [2026-10-19 19:40:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.8159 [2026-10-19 19:40:44][INFO][training.pretrain_engine] [Ph2|Step 25] loss=11.0274 | ppl=61539.99 | aux=0.03060 | lr=4.81e-05 | gnorm=1.766 | tok/s=29,239 | MFU=7.0% | step=143450ms | tokens=0.034B | mem=4.2/32.7GB | cost=$13.14/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 17:51 (142.2h) | Phase3: SKIPPED | SFT: Oct 25 22:36 (146.9h) | DPO (Final): Oct 25 23:11 (147.5h) [2026-10-19 19:40:44][INFO][training.pretrain_engine] Starting global step 26 (accumulating 32 micro-batches)... [2026-10-19 19:40:44][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 19:41:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.8637 [2026-10-19 19:41:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.8918 [2026-10-19 19:41:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.0254 [2026-10-19 19:41:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.8728 [2026-10-19 19:41:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.0744 [2026-10-19 19:41:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.5444 [2026-10-19 19:41:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.8104 [2026-10-19 19:41:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.9770 [2026-10-19 19:41:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.8366 [2026-10-19 19:41:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.9315 [2026-10-19 19:41:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.0384 [2026-10-19 19:41:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.0855 [2026-10-19 19:41:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.0505 [2026-10-19 19:41:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.8200 [2026-10-19 19:41:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.0521 [2026-10-19 19:41:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.0549 [2026-10-19 19:42:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.7591 [2026-10-19 19:42:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.8510 [2026-10-19 19:42:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.8728 [2026-10-19 19:42:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.9739 [2026-10-19 19:42:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.2306 [2026-10-19 19:42:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.2760 [2026-10-19 19:42:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.4668 [2026-10-19 19:42:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.2138 [2026-10-19 19:42:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.9738 [2026-10-19 19:42:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.6839 [2026-10-19 19:42:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.7694 [2026-10-19 19:42:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.9890 [2026-10-19 19:43:03][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,542 | MFU: 1.8% [2026-10-19 19:43:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.8804 [2026-10-19 19:43:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.9477 [2026-10-19 19:43:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.9195 [2026-10-19 19:43:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.0926 [2026-10-19 19:43:07][INFO][training.pretrain_engine] [Ph2|Step 26] loss=11.0222 | ppl=61221.06 | aux=0.03057 | lr=4.87e-05 | gnorm=1.766 | tok/s=29,238 | MFU=7.0% | step=143456ms | tokens=0.035B | mem=4.2/32.7GB | cost=$13.66/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 17:54 (142.2h) | Phase3: SKIPPED | SFT: Oct 25 22:39 (146.9h) | DPO (Final): Oct 25 23:13 (147.5h) [2026-10-19 19:43:07][INFO][training.pretrain_engine] Starting global step 27 (accumulating 32 micro-batches)... [2026-10-19 19:43:07][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 19:43:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.2200 [2026-10-19 19:43:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.2135 [2026-10-19 19:43:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.2298 [2026-10-19 19:43:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.3300 [2026-10-19 19:43:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.7495 [2026-10-19 19:43:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.7689 [2026-10-19 19:43:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.8654 [2026-10-19 19:43:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.6897 [2026-10-19 19:44:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.1359 [2026-10-19 19:44:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.1210 [2026-10-19 19:44:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.2566 [2026-10-19 19:44:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.1298 [2026-10-19 19:44:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.9822 [2026-10-19 19:44:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.8660 [2026-10-19 19:44:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.8540 [2026-10-19 19:44:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.8801 [2026-10-19 19:44:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.8569 [2026-10-19 19:44:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.7586 [2026-10-19 19:44:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.7756 [2026-10-19 19:44:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.7964 [2026-10-19 19:44:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.0856 [2026-10-19 19:44:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.1568 [2026-10-19 19:44:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.9119 [2026-10-19 19:44:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.0029 [2026-10-19 19:45:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.2948 [2026-10-19 19:45:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.4329 [2026-10-19 19:45:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.4605 [2026-10-19 19:45:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.0372 [2026-10-19 19:45:26][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,562 | MFU: 1.8% [2026-10-19 19:45:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.9243 [2026-10-19 19:45:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.7555 [2026-10-19 19:45:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.8978 [2026-10-19 19:45:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.7494 [2026-10-19 19:45:31][INFO][training.pretrain_engine] [Ph2|Step 27] loss=10.9948 | ppl=59561.12 | aux=0.03038 | lr=4.93e-05 | gnorm=1.664 | tok/s=29,240 | MFU=7.0% | step=143446ms | tokens=0.036B | mem=4.2/32.7GB | cost=$14.19/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 17:55 (142.2h) | Phase3: SKIPPED | SFT: Oct 25 22:40 (146.9h) | DPO (Final): Oct 25 23:14 (147.5h) [2026-10-19 19:45:34][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step27 | Tokens: 0.036B | Spent: $14.20 | Remaining: $385.80 [2026-10-19 19:45:34][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step27 [2026-10-19 19:45:34][INFO][training.pretrain_engine] Starting global step 28 (accumulating 32 micro-batches)... [2026-10-19 19:45:34][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 19:45:34][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-19 19:45:34][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-19 19:45:34][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-19 19:45:34][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step27 to Phase-Technologies/forge-500m-pretrain (path: phase2_step27)... [2026-10-19 19:45:34][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-19 19:45:44][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-19 19:45:44][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step27 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-19 19:45:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.8520 [2026-10-19 19:45:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.8473 [2026-10-19 19:45:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.1244 [2026-10-19 19:45:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.8385 [2026-10-19 19:46:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.0898 [2026-10-19 19:46:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.6822 [2026-10-19 19:46:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.7879 [2026-10-19 19:46:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.0628 [2026-10-19 19:46:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.4926 [2026-10-19 19:46:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.6238 [2026-10-19 19:46:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.6510 [2026-10-19 19:46:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.6849 [2026-10-19 19:46:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.7834 [2026-10-19 19:46:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.7377 [2026-10-19 19:46:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.7533 [2026-10-19 19:46:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.8350 [2026-10-19 19:46:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.5449 [2026-10-19 19:47:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.7701 [2026-10-19 19:47:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.6130 [2026-10-19 19:47:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.8302 [2026-10-19 19:47:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.8163 [2026-10-19 19:47:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.7982 [2026-10-19 19:47:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.9439 [2026-10-19 19:47:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.9093 [2026-10-19 19:47:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.9768 [2026-10-19 19:47:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.0322 [2026-10-19 19:47:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.0766 [2026-10-19 19:47:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.8720 [2026-10-19 19:47:53][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,554 | MFU: 1.8% [2026-10-19 19:47:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.1089 [2026-10-19 19:47:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.5860 [2026-10-19 19:47:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.0650 [2026-10-19 19:47:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.9020 [2026-10-19 19:47:57][INFO][training.pretrain_engine] [Ph2|Step 28] loss=11.0464 | ppl=62720.30 | aux=0.03028 | lr=4.98e-05 | gnorm=1.688 | tok/s=29,241 | MFU=7.0% | step=143442ms | tokens=0.037B | mem=4.2/32.7GB | cost=$14.72/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 17:56 (142.1h) | Phase3: SKIPPED | SFT: Oct 25 22:41 (146.9h) | DPO (Final): Oct 25 23:16 (147.5h) [2026-10-19 19:47:57][INFO][training.pretrain_engine] Starting global step 29 (accumulating 32 micro-batches)... [2026-10-19 19:47:57][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 19:48:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.7612 [2026-10-19 19:48:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.9531 [2026-10-19 19:48:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.8500 [2026-10-19 19:48:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.9771 [2026-10-19 19:48:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.4177 [2026-10-19 19:48:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.3864 [2026-10-19 19:48:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.4029 [2026-10-19 19:48:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.4851 [2026-10-19 19:48:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.6407 [2026-10-19 19:48:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.7858 [2026-10-19 19:48:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.6876 [2026-10-19 19:48:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.6138 [2026-10-19 19:49:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.7505 [2026-10-19 19:49:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.8643 [2026-10-19 19:49:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.9614 [2026-10-19 19:49:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.7783 [2026-10-19 19:49:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.9618 [2026-10-19 19:49:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.1019 [2026-10-19 19:49:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.0083 [2026-10-19 19:49:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.0511 [2026-10-19 19:49:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.5307 [2026-10-19 19:49:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.3188 [2026-10-19 19:49:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.3802 [2026-10-19 19:49:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.2844 [2026-10-19 19:50:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.8662 [2026-10-19 19:50:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.8313 [2026-10-19 19:50:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.0311 [2026-10-19 19:50:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.8017 [2026-10-19 19:50:16][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,562 | MFU: 1.8% [2026-10-19 19:50:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.5939 [2026-10-19 19:50:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.4041 [2026-10-19 19:50:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.6318 [2026-10-19 19:50:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.3605 [2026-10-19 19:50:20][INFO][training.pretrain_engine] [Ph2|Step 29] loss=11.0396 | ppl=62295.25 | aux=0.02999 | lr=5.04e-05 | gnorm=1.547 | tok/s=29,242 | MFU=7.0% | step=143433ms | tokens=0.038B | mem=4.2/32.7GB | cost=$15.25/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 17:58 (142.1h) | Phase3: SKIPPED | SFT: Oct 25 22:43 (146.9h) | DPO (Final): Oct 25 23:17 (147.4h) [2026-10-19 19:50:20][INFO][training.pretrain_engine] Starting global step 30 (accumulating 32 micro-batches)... [2026-10-19 19:50:20][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 19:50:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.9731 [2026-10-19 19:50:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.8203 [2026-10-19 19:50:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.8385 [2026-10-19 19:50:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.8269 [2026-10-19 19:50:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.3662 [2026-10-19 19:50:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.3267 [2026-10-19 19:50:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.2526 [2026-10-19 19:50:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.6906 [2026-10-19 19:51:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.8995 [2026-10-19 19:51:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.8719 [2026-10-19 19:51:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.9177 [2026-10-19 19:51:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.7655 [2026-10-19 19:51:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.2683 [2026-10-19 19:51:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.9401 [2026-10-19 19:51:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.1356 [2026-10-19 19:51:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.2226 [2026-10-19 19:51:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.8169 [2026-10-19 19:51:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.8957 [2026-10-19 19:51:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.8401 [2026-10-19 19:51:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.7900 [2026-10-19 19:52:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.7842 [2026-10-19 19:52:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.5233 [2026-10-19 19:52:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.6822 [2026-10-19 19:52:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.7827 [2026-10-19 19:52:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.0496 [2026-10-19 19:52:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.8506 [2026-10-19 19:52:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.0220 [2026-10-19 19:52:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.0360 [2026-10-19 19:52:40][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,541 | MFU: 1.8% [2026-10-19 19:52:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.1398 [2026-10-19 19:52:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.1358 [2026-10-19 19:52:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.9877 [2026-10-19 19:52:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.1783 [2026-10-19 19:52:44][INFO][training.pretrain_engine] [Ph2|Step 30] loss=10.9450 | ppl=56668.62 | aux=0.03012 | lr=5.10e-05 | gnorm=1.680 | tok/s=29,241 | MFU=7.0% | step=143438ms | tokens=0.039B | mem=4.2/32.7GB | cost=$15.77/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 18:00 (142.1h) | Phase3: SKIPPED | SFT: Oct 25 22:45 (146.9h) | DPO (Final): Oct 25 23:19 (147.4h) [2026-10-19 19:52:44][INFO][training.pretrain_engine] Starting global step 31 (accumulating 32 micro-batches)... [2026-10-19 19:52:44][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 19:53:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.5562 [2026-10-19 19:53:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.7247 [2026-10-19 19:53:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.9554 [2026-10-19 19:53:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.6407 [2026-10-19 19:53:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.0706 [2026-10-19 19:53:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.9560 [2026-10-19 19:53:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.2231 [2026-10-19 19:53:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.2684 [2026-10-19 19:53:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.0583 [2026-10-19 19:53:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.4123 [2026-10-19 19:53:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.6288 [2026-10-19 19:53:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.8245 [2026-10-19 19:53:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.0043 [2026-10-19 19:53:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.2957 [2026-10-19 19:53:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.1343 [2026-10-19 19:53:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.0347 [2026-10-19 19:54:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.1174 [2026-10-19 19:54:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.8997 [2026-10-19 19:54:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.8950 [2026-10-19 19:54:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.2116 [2026-10-19 19:54:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.6827 [2026-10-19 19:54:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.5462 [2026-10-19 19:54:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.5520 [2026-10-19 19:54:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.7261 [2026-10-19 19:54:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.0376 [2026-10-19 19:54:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.0360 [2026-10-19 19:54:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.1315 [2026-10-19 19:54:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.9736 [2026-10-19 19:55:03][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,551 | MFU: 1.8% [2026-10-19 19:55:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.8681 [2026-10-19 19:55:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.7834 [2026-10-19 19:55:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.8762 [2026-10-19 19:55:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.9168 [2026-10-19 19:55:07][INFO][training.pretrain_engine] [Ph2|Step 31] loss=10.8919 | ppl=53741.42 | aux=0.03004 | lr=5.15e-05 | gnorm=1.750 | tok/s=29,242 | MFU=7.0% | step=143435ms | tokens=0.040B | mem=4.2/32.7GB | cost=$16.30/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 18:01 (142.1h) | Phase3: SKIPPED | SFT: Oct 25 22:46 (146.9h) | DPO (Final): Oct 25 23:20 (147.4h) [2026-10-19 19:55:07][INFO][training.pretrain_engine] Starting global step 32 (accumulating 32 micro-batches)... [2026-10-19 19:55:07][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 19:55:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.0695 [2026-10-19 19:55:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.9810 [2026-10-19 19:55:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.9341 [2026-10-19 19:55:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.8613 [2026-10-19 19:55:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.7607 [2026-10-19 19:55:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.6042 [2026-10-19 19:55:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.6524 [2026-10-19 19:55:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.7593 [2026-10-19 19:56:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.9118 [2026-10-19 19:56:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.7500 [2026-10-19 19:56:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.8400 [2026-10-19 19:56:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.9432 [2026-10-19 19:56:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.9635 [2026-10-19 19:56:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.9557 [2026-10-19 19:56:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.7246 [2026-10-19 19:56:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.7891 [2026-10-19 19:56:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.5886 [2026-10-19 19:56:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.5195 [2026-10-19 19:56:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.4781 [2026-10-19 19:56:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.7599 [2026-10-19 19:56:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.8782 [2026-10-19 19:56:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.9475 [2026-10-19 19:56:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.3981 [2026-10-19 19:56:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.7242 [2026-10-19 19:57:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.6183 [2026-10-19 19:57:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.9861 [2026-10-19 19:57:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.7262 [2026-10-19 19:57:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.8220 [2026-10-19 19:57:26][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,553 | MFU: 1.8% [2026-10-19 19:57:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.9192 [2026-10-19 19:57:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.0276 [2026-10-19 19:57:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.8975 [2026-10-19 19:57:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.9906 [2026-10-19 19:57:31][INFO][training.pretrain_engine] [Ph2|Step 32] loss=10.8727 | ppl=52718.16 | aux=0.02998 | lr=5.21e-05 | gnorm=1.750 | tok/s=29,243 | MFU=7.0% | step=143431ms | tokens=0.041B | mem=4.2/32.7GB | cost=$16.82/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 18:03 (142.1h) | Phase3: SKIPPED | SFT: Oct 25 22:48 (146.8h) | DPO (Final): Oct 25 23:22 (147.4h) [2026-10-19 19:57:31][INFO][training.pretrain_engine] Starting global step 33 (accumulating 32 micro-batches)... [2026-10-19 19:57:31][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 19:57:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.8777 [2026-10-19 19:57:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.9297 [2026-10-19 19:57:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.6545 [2026-10-19 19:57:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.8090 [2026-10-19 19:58:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.8955 [2026-10-19 19:58:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.9216 [2026-10-19 19:58:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.8500 [2026-10-19 19:58:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.9526 [2026-10-19 19:58:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.7740 [2026-10-19 19:58:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.8809 [2026-10-19 19:58:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.7804 [2026-10-19 19:58:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.7461 [2026-10-19 19:58:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.9434 [2026-10-19 19:58:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.1763 [2026-10-19 19:58:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.9841 [2026-10-19 19:58:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.9824 [2026-10-19 19:59:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.8255 [2026-10-19 19:59:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.8632 [2026-10-19 19:59:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.2161 [2026-10-19 19:59:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.7980 [2026-10-19 19:59:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.5572 [2026-10-19 19:59:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.7592 [2026-10-19 19:59:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.5854 [2026-10-19 19:59:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.6272 [2026-10-19 19:59:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.9049 [2026-10-19 19:59:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.9655 [2026-10-19 19:59:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.0334 [2026-10-19 19:59:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.0595 [2026-10-19 19:59:50][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,538 | MFU: 1.8% [2026-10-19 19:59:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.2098 [2026-10-19 19:59:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.1998 [2026-10-19 19:59:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.1012 [2026-10-19 19:59:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.1656 [2026-10-19 19:59:54][INFO][training.pretrain_engine] [Ph2|Step 33] loss=10.8359 | ppl=50812.67 | aux=0.02977 | lr=5.26e-05 | gnorm=1.703 | tok/s=29,241 | MFU=7.0% | step=143438ms | tokens=0.042B | mem=4.2/32.7GB | cost=$17.34/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 18:05 (142.1h) | Phase3: SKIPPED | SFT: Oct 25 22:50 (146.8h) | DPO (Final): Oct 25 23:24 (147.4h) [2026-10-19 19:59:54][INFO][training.pretrain_engine] Starting global step 34 (accumulating 32 micro-batches)... [2026-10-19 19:59:54][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 20:00:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.8486 [2026-10-19 20:00:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.8508 [2026-10-19 20:00:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.6678 [2026-10-19 20:00:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.9151 [2026-10-19 20:00:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.9250 [2026-10-19 20:00:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.8746 [2026-10-19 20:00:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.9584 [2026-10-19 20:00:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.9491 [2026-10-19 20:00:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.8372 [2026-10-19 20:00:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.8319 [2026-10-19 20:00:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.8604 [2026-10-19 20:00:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.7248 [2026-10-19 20:01:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.0081 [2026-10-19 20:01:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.9429 [2026-10-19 20:01:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.9852 [2026-10-19 20:01:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.8825 [2026-10-19 20:01:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.0685 [2026-10-19 20:01:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.3293 [2026-10-19 20:01:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.8947 [2026-10-19 20:01:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.2230 [2026-10-19 20:01:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.0871 [2026-10-19 20:01:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.9122 [2026-10-19 20:01:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.8441 [2026-10-19 20:01:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.9046 [2026-10-19 20:01:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.9927 [2026-10-19 20:01:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.0529 [2026-10-19 20:01:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.8864 [2026-10-19 20:02:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.6634 [2026-10-19 20:02:13][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,553 | MFU: 1.8% [2026-10-19 20:02:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.3798 [2026-10-19 20:02:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.1649 [2026-10-19 20:02:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.3246 [2026-10-19 20:02:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.2820 [2026-10-19 20:02:18][INFO][training.pretrain_engine] [Ph2|Step 34] loss=10.9192 | ppl=55225.06 | aux=0.02951 | lr=5.32e-05 | gnorm=1.586 | tok/s=29,242 | MFU=7.0% | step=143434ms | tokens=0.043B | mem=4.2/32.7GB | cost=$17.87/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 18:07 (142.1h) | Phase3: SKIPPED | SFT: Oct 25 22:52 (146.8h) | DPO (Final): Oct 25 23:26 (147.4h) [2026-10-19 20:02:18][INFO][training.pretrain_engine] Starting global step 35 (accumulating 32 micro-batches)... [2026-10-19 20:02:18][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 20:02:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.9417 [2026-10-19 20:02:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.9197 [2026-10-19 20:02:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.9305 [2026-10-19 20:02:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.0999 [2026-10-19 20:02:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.6845 [2026-10-19 20:02:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.0932 [2026-10-19 20:02:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.9596 [2026-10-19 20:02:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.1403 [2026-10-19 20:03:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.7999 [2026-10-19 20:03:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.7305 [2026-10-19 20:03:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.8861 [2026-10-19 20:03:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.7073 [2026-10-19 20:03:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.6564 [2026-10-19 20:03:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.5554 [2026-10-19 20:03:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.8387 [2026-10-19 20:03:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.6221 [2026-10-19 20:03:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.8657 [2026-10-19 20:03:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.9435 [2026-10-19 20:03:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.9493 [2026-10-19 20:03:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.7499 [2026-10-19 20:04:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.2085 [2026-10-19 20:04:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.1072 [2026-10-19 20:04:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.9989 [2026-10-19 20:04:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.7199 [2026-10-19 20:04:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.1711 [2026-10-19 20:04:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.8922 [2026-10-19 20:04:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.0171 [2026-10-19 20:04:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.9094 [2026-10-19 20:04:37][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,539 | MFU: 1.8% [2026-10-19 20:04:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.5065 [2026-10-19 20:04:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.4189 [2026-10-19 20:04:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.5980 [2026-10-19 20:04:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.5897 [2026-10-19 20:04:41][INFO][training.pretrain_engine] [Ph2|Step 35] loss=10.8369 | ppl=50863.73 | aux=0.02937 | lr=5.38e-05 | gnorm=1.672 | tok/s=29,241 | MFU=7.0% | step=143438ms | tokens=0.044B | mem=4.2/32.7GB | cost=$18.39/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 18:09 (142.1h) | Phase3: SKIPPED | SFT: Oct 25 22:54 (146.8h) | DPO (Final): Oct 25 23:28 (147.4h) [2026-10-19 20:04:41][INFO][training.pretrain_engine] Starting global step 36 (accumulating 32 micro-batches)... [2026-10-19 20:04:41][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 20:04:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.0156 [2026-10-19 20:04:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.2141 [2026-10-19 20:04:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.0195 [2026-10-19 20:04:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.9686 [2026-10-19 20:05:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.0883 [2026-10-19 20:05:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.0652 [2026-10-19 20:05:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.0347 [2026-10-19 20:05:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.0845 [2026-10-19 20:05:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.5887 [2026-10-19 20:05:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.0896 [2026-10-19 20:05:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.3748 [2026-10-19 20:05:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.9303 [2026-10-19 20:05:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.6524 [2026-10-19 20:05:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.7915 [2026-10-19 20:05:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.7449 [2026-10-19 20:05:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.6106 [2026-10-19 20:06:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.9015 [2026-10-19 20:06:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.7485 [2026-10-19 20:06:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.0015 [2026-10-19 20:06:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.9285 [2026-10-19 20:06:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.9204 [2026-10-19 20:06:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.0731 [2026-10-19 20:06:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.0413 [2026-10-19 20:06:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.0991 [2026-10-19 20:06:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.7087 [2026-10-19 20:06:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.0007 [2026-10-19 20:06:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.7113 [2026-10-19 20:06:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.8964 [2026-10-19 20:07:00][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,539 | MFU: 1.8% [2026-10-19 20:07:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.5977 [2026-10-19 20:07:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.5968 [2026-10-19 20:07:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.7917 [2026-10-19 20:07:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.5490 [2026-10-19 20:07:05][INFO][training.pretrain_engine] [Ph2|Step 36] loss=10.9009 | ppl=54223.07 | aux=0.02927 | lr=5.43e-05 | gnorm=1.609 | tok/s=29,240 | MFU=7.0% | step=143443ms | tokens=0.045B | mem=4.2/32.7GB | cost=$18.92/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 18:11 (142.1h) | Phase3: SKIPPED | SFT: Oct 25 22:56 (146.8h) | DPO (Final): Oct 25 23:30 (147.4h) [2026-10-19 20:07:08][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step36 | Tokens: 0.045B | Spent: $18.93 | Remaining: $381.07 [2026-10-19 20:07:08][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step36 [2026-10-19 20:07:08][INFO][training.pretrain_engine] Starting global step 37 (accumulating 32 micro-batches)... [2026-10-19 20:07:08][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 20:07:08][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-19 20:07:08][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-19 20:07:08][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-19 20:07:08][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step36 to Phase-Technologies/forge-500m-pretrain (path: phase2_step36)... [2026-10-19 20:07:08][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-19 20:07:17][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-19 20:07:17][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step36 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-19 20:07:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.6451 [2026-10-19 20:07:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.1095 [2026-10-19 20:07:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.9472 [2026-10-19 20:07:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.7864 [2026-10-19 20:07:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.7535 [2026-10-19 20:07:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.9683 [2026-10-19 20:07:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.7620 [2026-10-19 20:07:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.7784 [2026-10-19 20:07:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.8899 [2026-10-19 20:07:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.6570 [2026-10-19 20:07:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.4277 [2026-10-19 20:08:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.7796 [2026-10-19 20:08:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.5689 [2026-10-19 20:08:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.5933 [2026-10-19 20:08:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.6023 [2026-10-19 20:08:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.5365 [2026-10-19 20:08:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.6946 [2026-10-19 20:08:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.8554 [2026-10-19 20:08:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.6149 [2026-10-19 20:08:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.8806 [2026-10-19 20:08:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.0848 [2026-10-19 20:08:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.7404 [2026-10-19 20:08:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.7777 [2026-10-19 20:08:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.0030 [2026-10-19 20:09:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.8763 [2026-10-19 20:09:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.1083 [2026-10-19 20:09:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.7651 [2026-10-19 20:09:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.1315 [2026-10-19 20:09:27][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,552 | MFU: 1.8% [2026-10-19 20:09:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.4020 [2026-10-19 20:09:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.4760 [2026-10-19 20:09:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.4631 [2026-10-19 20:09:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.3985 [2026-10-19 20:09:32][INFO][training.pretrain_engine] [Ph2|Step 37] loss=10.8483 | ppl=51444.97 | aux=0.02933 | lr=5.49e-05 | gnorm=1.680 | tok/s=29,241 | MFU=7.0% | step=143441ms | tokens=0.046B | mem=4.2/32.7GB | cost=$19.45/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 18:12 (142.1h) | Phase3: SKIPPED | SFT: Oct 25 22:57 (146.8h) | DPO (Final): Oct 25 23:32 (147.4h) [2026-10-19 20:09:32][INFO][training.pretrain_engine] Starting global step 38 (accumulating 32 micro-batches)... [2026-10-19 20:09:32][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 20:09:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.8041 [2026-10-19 20:09:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.8368 [2026-10-19 20:09:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.7405 [2026-10-19 20:09:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.8512 [2026-10-19 20:10:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.7775 [2026-10-19 20:10:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.7665 [2026-10-19 20:10:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.8554 [2026-10-19 20:10:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.1105 [2026-10-19 20:10:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.8767 [2026-10-19 20:10:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.0265 [2026-10-19 20:10:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.9611 [2026-10-19 20:10:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.0535 [2026-10-19 20:10:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.6381 [2026-10-19 20:10:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.6802 [2026-10-19 20:10:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.7357 [2026-10-19 20:10:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.7186 [2026-10-19 20:11:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.8586 [2026-10-19 20:11:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.0423 [2026-10-19 20:11:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.9097 [2026-10-19 20:11:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.1127 [2026-10-19 20:11:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.0681 [2026-10-19 20:11:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.8935 [2026-10-19 20:11:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.1688 [2026-10-19 20:11:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.9180 [2026-10-19 20:11:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.8663 [2026-10-19 20:11:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.9937 [2026-10-19 20:11:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.0061 [2026-10-19 20:11:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.7729 [2026-10-19 20:11:50][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,546 | MFU: 1.8% [2026-10-19 20:11:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.7253 [2026-10-19 20:11:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.7934 [2026-10-19 20:11:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.0253 [2026-10-19 20:11:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.1238 [2026-10-19 20:11:55][INFO][training.pretrain_engine] [Ph2|Step 38] loss=10.8298 | ppl=50505.26 | aux=0.02903 | lr=5.55e-05 | gnorm=1.602 | tok/s=29,241 | MFU=7.0% | step=143441ms | tokens=0.047B | mem=4.2/32.7GB | cost=$19.98/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 18:14 (142.0h) | Phase3: SKIPPED | SFT: Oct 25 22:59 (146.8h) | DPO (Final): Oct 25 23:33 (147.4h) [2026-10-19 20:11:55][INFO][training.pretrain_engine] Starting global step 39 (accumulating 32 micro-batches)... [2026-10-19 20:11:55][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 20:12:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.9582 [2026-10-19 20:12:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.0047 [2026-10-19 20:12:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.7924 [2026-10-19 20:12:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.8072 [2026-10-19 20:12:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.4921 [2026-10-19 20:12:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.7720 [2026-10-19 20:12:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.9372 [2026-10-19 20:12:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.8477 [2026-10-19 20:12:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.6940 [2026-10-19 20:12:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.7102 [2026-10-19 20:12:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.8742 [2026-10-19 20:12:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.7134 [2026-10-19 20:13:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.8432 [2026-10-19 20:13:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.7897 [2026-10-19 20:13:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.9366 [2026-10-19 20:13:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.8555 [2026-10-19 20:13:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.6956 [2026-10-19 20:13:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.7064 [2026-10-19 20:13:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.7733 [2026-10-19 20:13:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.0024 [2026-10-19 20:13:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.5587 [2026-10-19 20:13:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.7507 [2026-10-19 20:13:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.4406 [2026-10-19 20:13:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.3662 [2026-10-19 20:14:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.4990 [2026-10-19 20:14:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.6277 [2026-10-19 20:14:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.9180 [2026-10-19 20:14:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.8662 [2026-10-19 20:14:14][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,545 | MFU: 1.8% [2026-10-19 20:14:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.3420 [2026-10-19 20:14:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.4115 [2026-10-19 20:14:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.4831 [2026-10-19 20:14:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.3299 [2026-10-19 20:14:18][INFO][training.pretrain_engine] [Ph2|Step 39] loss=10.8179 | ppl=49907.85 | aux=0.02899 | lr=5.60e-05 | gnorm=1.586 | tok/s=29,240 | MFU=7.0% | step=143442ms | tokens=0.048B | mem=4.2/32.7GB | cost=$20.50/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 18:16 (142.0h) | Phase3: SKIPPED | SFT: Oct 25 23:01 (146.8h) | DPO (Final): Oct 25 23:35 (147.4h) [2026-10-19 20:14:18][INFO][training.pretrain_engine] Starting global step 40 (accumulating 32 micro-batches)... [2026-10-19 20:14:18][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 20:14:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.6851 [2026-10-19 20:14:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.4319 [2026-10-19 20:14:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.5797 [2026-10-19 20:14:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.5326 [2026-10-19 20:14:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.5604 [2026-10-19 20:14:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.2169 [2026-10-19 20:14:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.3795 [2026-10-19 20:14:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.5836 [2026-10-19 20:15:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.2768 [2026-10-19 20:15:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.3752 [2026-10-19 20:15:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.7385 [2026-10-19 20:15:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.1212 [2026-10-19 20:15:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.3957 [2026-10-19 20:15:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.4519 [2026-10-19 20:15:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.4012 [2026-10-19 20:15:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.5150 [2026-10-19 20:15:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.8328 [2026-10-19 20:15:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.6068 [2026-10-19 20:15:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.7207 [2026-10-19 20:15:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.7313 [2026-10-19 20:16:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.8082 [2026-10-19 20:16:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.9345 [2026-10-19 20:16:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.8743 [2026-10-19 20:16:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.8437 [2026-10-19 20:16:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.8053 [2026-10-19 20:16:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.8387 [2026-10-19 20:16:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.9695 [2026-10-19 20:16:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.8328 [2026-10-19 20:16:37][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,580 | MFU: 1.8% [2026-10-19 20:16:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.5266 [2026-10-19 20:16:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.4502 [2026-10-19 20:16:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.5695 [2026-10-19 20:16:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.4140 [2026-10-19 20:16:41][INFO][training.pretrain_engine] [Ph2|Step 40] loss=10.7150 | ppl=45025.55 | aux=0.02914 | lr=5.66e-05 | gnorm=1.719 | tok/s=29,243 | MFU=7.0% | step=143430ms | tokens=0.049B | mem=4.2/32.7GB | cost=$21.03/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 18:17 (142.0h) | Phase3: SKIPPED | SFT: Oct 25 23:02 (146.8h) | DPO (Final): Oct 25 23:36 (147.3h) [2026-10-19 20:16:41][INFO][training.pretrain_engine] Starting global step 41 (accumulating 32 micro-batches)... [2026-10-19 20:16:41][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 20:16:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.5463 [2026-10-19 20:16:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.6111 [2026-10-19 20:16:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.6565 [2026-10-19 20:16:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.4590 [2026-10-19 20:17:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.5431 [2026-10-19 20:17:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.4157 [2026-10-19 20:17:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.7874 [2026-10-19 20:17:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.3683 [2026-10-19 20:17:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.8028 [2026-10-19 20:17:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.7641 [2026-10-19 20:17:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.9113 [2026-10-19 20:17:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.7957 [2026-10-19 20:17:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.1916 [2026-10-19 20:17:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.1588 [2026-10-19 20:17:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.9122 [2026-10-19 20:17:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.1062 [2026-10-19 20:18:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.6937 [2026-10-19 20:18:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.5260 [2026-10-19 20:18:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.4263 [2026-10-19 20:18:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.4803 [2026-10-19 20:18:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.4494 [2026-10-19 20:18:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.3571 [2026-10-19 20:18:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.3846 [2026-10-19 20:18:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.4414 [2026-10-19 20:18:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.7079 [2026-10-19 20:18:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.9576 [2026-10-19 20:18:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.0716 [2026-10-19 20:18:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.1600 [2026-10-19 20:19:00][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,575 | MFU: 1.8% [2026-10-19 20:19:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.6267 [2026-10-19 20:19:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.7039 [2026-10-19 20:19:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.7435 [2026-10-19 20:19:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.9487 [2026-10-19 20:19:04][INFO][training.pretrain_engine] [Ph2|Step 41] loss=10.6984 | ppl=44285.36 | aux=0.02893 | lr=5.72e-05 | gnorm=1.703 | tok/s=29,245 | MFU=7.0% | step=143418ms | tokens=0.050B | mem=4.2/32.7GB | cost=$21.55/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 18:18 (142.0h) | Phase3: SKIPPED | SFT: Oct 25 23:03 (146.7h) | DPO (Final): Oct 25 23:37 (147.3h) [2026-10-19 20:19:04][INFO][training.pretrain_engine] Starting global step 42 (accumulating 32 micro-batches)... [2026-10-19 20:19:04][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 20:19:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.8650 [2026-10-19 20:19:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.0996 [2026-10-19 20:19:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=11.0396 [2026-10-19 20:19:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.6589 [2026-10-19 20:19:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.1353 [2026-10-19 20:19:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.8409 [2026-10-19 20:19:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.8243 [2026-10-19 20:19:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.9142 [2026-10-19 20:19:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.4625 [2026-10-19 20:19:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.4361 [2026-10-19 20:19:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.4167 [2026-10-19 20:19:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.4769 [2026-10-19 20:20:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.7145 [2026-10-19 20:20:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.5380 [2026-10-19 20:20:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.4665 [2026-10-19 20:20:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.8417 [2026-10-19 20:20:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.6600 [2026-10-19 20:20:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.5562 [2026-10-19 20:20:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.7291 [2026-10-19 20:20:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.7773 [2026-10-19 20:20:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.0002 [2026-10-19 20:20:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.1670 [2026-10-19 20:20:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=11.1863 [2026-10-19 20:20:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.9548 [2026-10-19 20:21:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.7588 [2026-10-19 20:21:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.8172 [2026-10-19 20:21:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.8618 [2026-10-19 20:21:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.8968 [2026-10-19 20:21:23][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,580 | MFU: 1.8% [2026-10-19 20:21:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.6984 [2026-10-19 20:21:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.6471 [2026-10-19 20:21:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.5698 [2026-10-19 20:21:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.5567 [2026-10-19 20:21:27][INFO][training.pretrain_engine] [Ph2|Step 42] loss=10.7614 | ppl=47166.66 | aux=0.02871 | lr=5.77e-05 | gnorm=1.578 | tok/s=29,248 | MFU=7.0% | step=143406ms | tokens=0.051B | mem=4.2/32.7GB | cost=$22.07/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 18:19 (142.0h) | Phase3: SKIPPED | SFT: Oct 25 23:04 (146.7h) | DPO (Final): Oct 25 23:38 (147.3h) [2026-10-19 20:21:27][INFO][training.pretrain_engine] Starting global step 43 (accumulating 32 micro-batches)... [2026-10-19 20:21:27][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 20:21:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.7440 [2026-10-19 20:21:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.8463 [2026-10-19 20:21:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.5153 [2026-10-19 20:21:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.5203 [2026-10-19 20:22:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.5625 [2026-10-19 20:22:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.6939 [2026-10-19 20:22:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.8603 [2026-10-19 20:22:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.6223 [2026-10-19 20:22:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.5841 [2026-10-19 20:22:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.8642 [2026-10-19 20:22:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.5547 [2026-10-19 20:22:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.5982 [2026-10-19 20:22:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.6587 [2026-10-19 20:22:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.7963 [2026-10-19 20:22:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.6623 [2026-10-19 20:22:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.6237 [2026-10-19 20:22:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.7750 [2026-10-19 20:22:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.2884 [2026-10-19 20:22:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.5808 [2026-10-19 20:22:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.8579 [2026-10-19 20:23:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.5616 [2026-10-19 20:23:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.4045 [2026-10-19 20:23:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.4058 [2026-10-19 20:23:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.5372 [2026-10-19 20:23:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.7045 [2026-10-19 20:23:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=11.1281 [2026-10-19 20:23:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.8011 [2026-10-19 20:23:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.9568 [2026-10-19 20:23:45][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,592 | MFU: 1.8% [2026-10-19 20:23:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.8721 [2026-10-19 20:23:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.8306 [2026-10-19 20:23:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.0180 [2026-10-19 20:23:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.0251 [2026-10-19 20:23:50][INFO][training.pretrain_engine] [Ph2|Step 43] loss=10.7835 | ppl=48216.25 | aux=0.02842 | lr=5.83e-05 | gnorm=1.500 | tok/s=29,252 | MFU=7.0% | step=143387ms | tokens=0.052B | mem=4.2/32.7GB | cost=$22.59/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 18:20 (141.9h) | Phase3: SKIPPED | SFT: Oct 25 23:05 (146.7h) | DPO (Final): Oct 25 23:39 (147.3h) [2026-10-19 20:23:50][INFO][training.pretrain_engine] Starting global step 44 (accumulating 32 micro-batches)... [2026-10-19 20:23:50][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 20:24:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.8222 [2026-10-19 20:24:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.7023 [2026-10-19 20:24:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.5885 [2026-10-19 20:24:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.7417 [2026-10-19 20:24:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.7200 [2026-10-19 20:24:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.5017 [2026-10-19 20:24:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.5155 [2026-10-19 20:24:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.5337 [2026-10-19 20:24:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.0944 [2026-10-19 20:24:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.7663 [2026-10-19 20:24:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.0279 [2026-10-19 20:24:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.9120 [2026-10-19 20:25:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.4906 [2026-10-19 20:25:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.8879 [2026-10-19 20:25:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.0447 [2026-10-19 20:25:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.8096 [2026-10-19 20:25:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.5737 [2026-10-19 20:25:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.6376 [2026-10-19 20:25:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.3518 [2026-10-19 20:25:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.6584 [2026-10-19 20:25:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.4760 [2026-10-19 20:25:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.4296 [2026-10-19 20:25:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.4556 [2026-10-19 20:25:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.9662 [2026-10-19 20:25:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.6921 [2026-10-19 20:25:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.5851 [2026-10-19 20:25:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.5926 [2026-10-19 20:25:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.6816 [2026-10-19 20:26:08][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,587 | MFU: 1.8% [2026-10-19 20:26:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.4634 [2026-10-19 20:26:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.3597 [2026-10-19 20:26:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.3676 [2026-10-19 20:26:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.2582 [2026-10-19 20:26:13][INFO][training.pretrain_engine] [Ph2|Step 44] loss=10.7203 | ppl=45265.65 | aux=0.02841 | lr=5.89e-05 | gnorm=1.570 | tok/s=29,255 | MFU=7.0% | step=143372ms | tokens=0.053B | mem=4.2/32.7GB | cost=$23.11/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 18:21 (141.9h) | Phase3: SKIPPED | SFT: Oct 25 23:06 (146.7h) | DPO (Final): Oct 25 23:40 (147.2h) [2026-10-19 20:26:13][INFO][training.pretrain_engine] Starting global step 45 (accumulating 32 micro-batches)... [2026-10-19 20:26:13][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 20:26:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.3709 [2026-10-19 20:26:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.6888 [2026-10-19 20:26:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.8494 [2026-10-19 20:26:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.4860 [2026-10-19 20:26:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.5054 [2026-10-19 20:26:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.5767 [2026-10-19 20:26:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.6213 [2026-10-19 20:26:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.6813 [2026-10-19 20:27:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.5021 [2026-10-19 20:27:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.4203 [2026-10-19 20:27:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.3673 [2026-10-19 20:27:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.4816 [2026-10-19 20:27:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.4498 [2026-10-19 20:27:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.5088 [2026-10-19 20:27:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.4472 [2026-10-19 20:27:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.4843 [2026-10-19 20:27:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.8593 [2026-10-19 20:27:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.7355 [2026-10-19 20:27:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.1596 [2026-10-19 20:27:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=11.0088 [2026-10-19 20:27:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.8886 [2026-10-19 20:27:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.6051 [2026-10-19 20:27:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.5262 [2026-10-19 20:28:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.7629 [2026-10-19 20:28:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.4414 [2026-10-19 20:28:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.5531 [2026-10-19 20:28:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.6926 [2026-10-19 20:28:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.7791 [2026-10-19 20:28:31][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,588 | MFU: 1.8% [2026-10-19 20:28:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.7154 [2026-10-19 20:28:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.6280 [2026-10-19 20:28:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.5521 [2026-10-19 20:28:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.5313 [2026-10-19 20:28:35][INFO][training.pretrain_engine] [Ph2|Step 45] loss=10.6535 | ppl=42339.95 | aux=0.02818 | lr=5.94e-05 | gnorm=1.570 | tok/s=29,258 | MFU=7.0% | step=143357ms | tokens=0.055B | mem=4.2/32.7GB | cost=$23.64/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 18:22 (141.9h) | Phase3: SKIPPED | SFT: Oct 25 23:07 (146.6h) | DPO (Final): Oct 25 23:41 (147.2h) [2026-10-19 20:28:39][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step45 | Tokens: 0.055B | Spent: $23.65 | Remaining: $376.35 [2026-10-19 20:28:39][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step45 [2026-10-19 20:28:39][INFO][training.pretrain_engine] Starting global step 46 (accumulating 32 micro-batches)... [2026-10-19 20:28:39][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 20:28:39][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-19 20:28:39][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-19 20:28:39][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-19 20:28:39][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step45 to Phase-Technologies/forge-500m-pretrain (path: phase2_step45)... [2026-10-19 20:28:39][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-19 20:28:48][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-19 20:28:48][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step45 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-19 20:28:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.7198 [2026-10-19 20:28:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.7700 [2026-10-19 20:28:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.5648 [2026-10-19 20:28:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.4467 [2026-10-19 20:29:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.5844 [2026-10-19 20:29:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.8342 [2026-10-19 20:29:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.4177 [2026-10-19 20:29:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.5942 [2026-10-19 20:29:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.5501 [2026-10-19 20:29:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.7584 [2026-10-19 20:29:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.7725 [2026-10-19 20:29:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.6522 [2026-10-19 20:29:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.4483 [2026-10-19 20:29:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.5804 [2026-10-19 20:29:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.5078 [2026-10-19 20:29:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.4631 [2026-10-19 20:30:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.5248 [2026-10-19 20:30:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.6352 [2026-10-19 20:30:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.4268 [2026-10-19 20:30:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.4674 [2026-10-19 20:30:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.3081 [2026-10-19 20:30:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.8167 [2026-10-19 20:30:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.3416 [2026-10-19 20:30:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.5414 [2026-10-19 20:30:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.7969 [2026-10-19 20:30:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.4795 [2026-10-19 20:30:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.5790 [2026-10-19 20:30:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.6396 [2026-10-19 20:30:57][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,584 | MFU: 1.8% [2026-10-19 20:31:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.5052 [2026-10-19 20:31:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.5708 [2026-10-19 20:31:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.5368 [2026-10-19 20:31:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.5788 [2026-10-19 20:31:02][INFO][training.pretrain_engine] [Ph2|Step 46] loss=10.5926 | ppl=39837.14 | aux=0.02812 | lr=6.00e-05 | gnorm=1.586 | tok/s=29,260 | MFU=7.0% | step=143347ms | tokens=0.056B | mem=4.2/32.7GB | cost=$24.17/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 18:23 (141.9h) | Phase3: SKIPPED | SFT: Oct 25 23:08 (146.6h) | DPO (Final): Oct 25 23:42 (147.2h) [2026-10-19 20:31:02][INFO][training.pretrain_engine] Starting global step 47 (accumulating 32 micro-batches)... [2026-10-19 20:31:02][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 20:31:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.4634 [2026-10-19 20:31:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.4424 [2026-10-19 20:31:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.7713 [2026-10-19 20:31:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.6650 [2026-10-19 20:31:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.5675 [2026-10-19 20:31:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.5963 [2026-10-19 20:31:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.6484 [2026-10-19 20:31:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.7639 [2026-10-19 20:31:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.9253 [2026-10-19 20:31:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.5613 [2026-10-19 20:31:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=11.0216 [2026-10-19 20:31:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.6717 [2026-10-19 20:32:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.4066 [2026-10-19 20:32:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.5219 [2026-10-19 20:32:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.6095 [2026-10-19 20:32:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.4722 [2026-10-19 20:32:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.9896 [2026-10-19 20:32:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.2889 [2026-10-19 20:32:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.6132 [2026-10-19 20:32:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.9492 [2026-10-19 20:32:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.1901 [2026-10-19 20:32:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.2640 [2026-10-19 20:32:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.4400 [2026-10-19 20:32:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.4148 [2026-10-19 20:33:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.0403 [2026-10-19 20:33:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.7096 [2026-10-19 20:33:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.6029 [2026-10-19 20:33:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.3558 [2026-10-19 20:33:20][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,583 | MFU: 1.8% [2026-10-19 20:33:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.3657 [2026-10-19 20:33:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.4783 [2026-10-19 20:33:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.7028 [2026-10-19 20:33:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.2859 [2026-10-19 20:33:24][INFO][training.pretrain_engine] [Ph2|Step 47] loss=10.6171 | ppl=40826.57 | aux=0.02791 | lr=6.06e-05 | gnorm=1.562 | tok/s=29,262 | MFU=7.0% | step=143335ms | tokens=0.057B | mem=4.2/32.7GB | cost=$24.69/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 18:24 (141.9h) | Phase3: SKIPPED | SFT: Oct 25 23:09 (146.6h) | DPO (Final): Oct 25 23:43 (147.2h) [2026-10-19 20:33:24][INFO][training.pretrain_engine] Starting global step 48 (accumulating 32 micro-batches)... [2026-10-19 20:33:24][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 20:33:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.9188 [2026-10-19 20:33:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.9111 [2026-10-19 20:33:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.8843 [2026-10-19 20:33:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.9812 [2026-10-19 20:34:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.7319 [2026-10-19 20:34:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.7309 [2026-10-19 20:34:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.8255 [2026-10-19 20:34:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.5561 [2026-10-19 20:34:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.3524 [2026-10-19 20:34:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.4838 [2026-10-19 20:34:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.5284 [2026-10-19 20:34:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.5009 [2026-10-19 20:34:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.3543 [2026-10-19 20:34:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.9637 [2026-10-19 20:34:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.1234 [2026-10-19 20:34:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.2086 [2026-10-19 20:34:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.2739 [2026-10-19 20:34:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.5233 [2026-10-19 20:34:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.5415 [2026-10-19 20:34:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.4585 [2026-10-19 20:35:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.7008 [2026-10-19 20:35:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.6630 [2026-10-19 20:35:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.7298 [2026-10-19 20:35:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.8599 [2026-10-19 20:35:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.4356 [2026-10-19 20:35:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.3933 [2026-10-19 20:35:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.4461 [2026-10-19 20:35:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.3744 [2026-10-19 20:35:42][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,588 | MFU: 1.8% [2026-10-19 20:35:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.0719 [2026-10-19 20:35:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.9758 [2026-10-19 20:35:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.0533 [2026-10-19 20:35:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.0718 [2026-10-19 20:35:47][INFO][training.pretrain_engine] [Ph2|Step 48] loss=10.6296 | ppl=41342.35 | aux=0.02780 | lr=6.11e-05 | gnorm=1.531 | tok/s=29,265 | MFU=7.0% | step=143321ms | tokens=0.058B | mem=4.2/32.7GB | cost=$25.21/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 18:25 (141.8h) | Phase3: SKIPPED | SFT: Oct 25 23:10 (146.6h) | DPO (Final): Oct 25 23:44 (147.1h) [2026-10-19 20:35:47][INFO][training.pretrain_engine] Starting global step 49 (accumulating 32 micro-batches)... [2026-10-19 20:35:47][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 20:36:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.7275 [2026-10-19 20:36:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.8971 [2026-10-19 20:36:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.8656 [2026-10-19 20:36:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.8953 [2026-10-19 20:36:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.4277 [2026-10-19 20:36:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.3422 [2026-10-19 20:36:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.5413 [2026-10-19 20:36:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.6515 [2026-10-19 20:36:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.2276 [2026-10-19 20:36:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.4773 [2026-10-19 20:36:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.3176 [2026-10-19 20:36:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.6200 [2026-10-19 20:36:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.5823 [2026-10-19 20:36:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.3559 [2026-10-19 20:36:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.2953 [2026-10-19 20:36:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.7222 [2026-10-19 20:37:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.4900 [2026-10-19 20:37:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.4504 [2026-10-19 20:37:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.5479 [2026-10-19 20:37:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.6267 [2026-10-19 20:37:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.4825 [2026-10-19 20:37:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.5667 [2026-10-19 20:37:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.6063 [2026-10-19 20:37:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.7106 [2026-10-19 20:37:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.3775 [2026-10-19 20:37:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.3252 [2026-10-19 20:37:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.3367 [2026-10-19 20:37:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.5305 [2026-10-19 20:38:05][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,588 | MFU: 1.8% [2026-10-19 20:38:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.2983 [2026-10-19 20:38:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.8854 [2026-10-19 20:38:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.6855 [2026-10-19 20:38:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=11.1802 [2026-10-19 20:38:10][INFO][training.pretrain_engine] [Ph2|Step 49] loss=10.5748 | ppl=39134.99 | aux=0.02766 | lr=6.17e-05 | gnorm=1.539 | tok/s=29,267 | MFU=7.0% | step=143314ms | tokens=0.059B | mem=4.2/32.7GB | cost=$25.74/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 18:26 (141.8h) | Phase3: SKIPPED | SFT: Oct 25 23:11 (146.6h) | DPO (Final): Oct 25 23:45 (147.1h) [2026-10-19 20:38:10][INFO][training.pretrain_engine] Starting global step 50 (accumulating 32 micro-batches)... [2026-10-19 20:38:10][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 20:38:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.2147 [2026-10-19 20:38:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.3188 [2026-10-19 20:38:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.5006 [2026-10-19 20:38:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.2321 [2026-10-19 20:38:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.6150 [2026-10-19 20:38:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.3371 [2026-10-19 20:38:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.3480 [2026-10-19 20:38:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.6214 [2026-10-19 20:39:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.5769 [2026-10-19 20:39:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.5659 [2026-10-19 20:39:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.5932 [2026-10-19 20:39:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.7498 [2026-10-19 20:39:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.5897 [2026-10-19 20:39:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.6227 [2026-10-19 20:39:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.5671 [2026-10-19 20:39:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.2070 [2026-10-19 20:39:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.2150 [2026-10-19 20:39:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.2323 [2026-10-19 20:39:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.4910 [2026-10-19 20:39:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.3587 [2026-10-19 20:39:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.3999 [2026-10-19 20:39:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.4808 [2026-10-19 20:39:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.4453 [2026-10-19 20:39:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.3453 [2026-10-19 20:40:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.3647 [2026-10-19 20:40:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.1811 [2026-10-19 20:40:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.3548 [2026-10-19 20:40:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.3779 [2026-10-19 20:40:28][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,610 | MFU: 1.8% [2026-10-19 20:40:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.3206 [2026-10-19 20:40:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.4127 [2026-10-19 20:40:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.4080 [2026-10-19 20:40:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.3092 [2026-10-19 20:40:33][INFO][training.pretrain_engine] [Ph2|Step 50] loss=10.5188 | ppl=37006.40 | aux=0.02747 | lr=6.23e-05 | gnorm=1.570 | tok/s=29,269 | MFU=7.0% | step=143302ms | tokens=0.060B | mem=4.2/32.7GB | cost=$26.26/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 18:27 (141.8h) | Phase3: SKIPPED | SFT: Oct 25 23:12 (146.5h) | DPO (Final): Oct 25 23:46 (147.1h) [2026-10-19 20:40:33][INFO][training.pretrain_engine] Starting global step 51 (accumulating 32 micro-batches)... [2026-10-19 20:40:33][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 20:40:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.5588 [2026-10-19 20:40:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.4442 [2026-10-19 20:40:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.3278 [2026-10-19 20:40:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.4461 [2026-10-19 20:41:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.5536 [2026-10-19 20:41:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.6336 [2026-10-19 20:41:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.5485 [2026-10-19 20:41:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.6648 [2026-10-19 20:41:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.6867 [2026-10-19 20:41:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.8455 [2026-10-19 20:41:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.9975 [2026-10-19 20:41:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.5889 [2026-10-19 20:41:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.7647 [2026-10-19 20:41:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.6559 [2026-10-19 20:41:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.8248 [2026-10-19 20:41:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.8513 [2026-10-19 20:42:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.3085 [2026-10-19 20:42:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.3382 [2026-10-19 20:42:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.3046 [2026-10-19 20:42:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.3899 [2026-10-19 20:42:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.2408 [2026-10-19 20:42:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.0890 [2026-10-19 20:42:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.1881 [2026-10-19 20:42:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.1385 [2026-10-19 20:42:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.3030 [2026-10-19 20:42:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.3036 [2026-10-19 20:42:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.1499 [2026-10-19 20:42:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.2959 [2026-10-19 20:42:50][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,623 | MFU: 1.8% [2026-10-19 20:42:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.3748 [2026-10-19 20:42:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.7171 [2026-10-19 20:42:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.4828 [2026-10-19 20:42:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.8383 [2026-10-19 20:42:56][INFO][training.pretrain_engine] [Ph2|Step 51] loss=10.5380 | ppl=37722.79 | aux=0.02740 | lr=6.28e-05 | gnorm=1.508 | tok/s=29,271 | MFU=7.0% | step=143294ms | tokens=0.061B | mem=4.2/32.7GB | cost=$26.78/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 18:29 (141.8h) | Phase3: SKIPPED | SFT: Oct 25 23:13 (146.5h) | DPO (Final): Oct 25 23:48 (147.1h) [2026-10-19 20:42:56][INFO][training.pretrain_engine] Starting global step 52 (accumulating 32 micro-batches)... [2026-10-19 20:42:56][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 20:43:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.4114 [2026-10-19 20:43:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.3997 [2026-10-19 20:43:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.5356 [2026-10-19 20:43:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.3684 [2026-10-19 20:43:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.7469 [2026-10-19 20:43:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.3857 [2026-10-19 20:43:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.3079 [2026-10-19 20:43:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.6910 [2026-10-19 20:43:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.4794 [2026-10-19 20:43:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.3667 [2026-10-19 20:43:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.4368 [2026-10-19 20:43:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.3664 [2026-10-19 20:44:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.8101 [2026-10-19 20:44:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.2970 [2026-10-19 20:44:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.4572 [2026-10-19 20:44:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.6535 [2026-10-19 20:44:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.5278 [2026-10-19 20:44:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.3569 [2026-10-19 20:44:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.4186 [2026-10-19 20:44:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.2408 [2026-10-19 20:44:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.6178 [2026-10-19 20:44:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.2781 [2026-10-19 20:44:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.2155 [2026-10-19 20:44:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.1786 [2026-10-19 20:45:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.5566 [2026-10-19 20:45:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.4051 [2026-10-19 20:45:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.5087 [2026-10-19 20:45:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.4479 [2026-10-19 20:45:13][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,608 | MFU: 1.8% [2026-10-19 20:45:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.1722 [2026-10-19 20:45:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.1320 [2026-10-19 20:45:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.2954 [2026-10-19 20:45:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.2945 [2026-10-19 20:45:18][INFO][training.pretrain_engine] [Ph2|Step 52] loss=10.4690 | ppl=35207.60 | aux=0.02730 | lr=6.34e-05 | gnorm=1.516 | tok/s=29,273 | MFU=7.0% | step=143281ms | tokens=0.062B | mem=4.2/32.7GB | cost=$27.30/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 18:30 (141.7h) | Phase3: SKIPPED | SFT: Oct 25 23:14 (146.5h) | DPO (Final): Oct 25 23:49 (147.1h) [2026-10-19 20:45:18][INFO][training.pretrain_engine] Starting global step 53 (accumulating 32 micro-batches)... [2026-10-19 20:45:18][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 20:45:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.3037 [2026-10-19 20:45:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.6494 [2026-10-19 20:45:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.3288 [2026-10-19 20:45:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.7283 [2026-10-19 20:45:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.0069 [2026-10-19 20:45:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.4541 [2026-10-19 20:45:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.2597 [2026-10-19 20:45:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.1898 [2026-10-19 20:46:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.4002 [2026-10-19 20:46:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.5122 [2026-10-19 20:46:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.2248 [2026-10-19 20:46:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.2379 [2026-10-19 20:46:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.2096 [2026-10-19 20:46:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.3638 [2026-10-19 20:46:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.1914 [2026-10-19 20:46:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.3460 [2026-10-19 20:46:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.2471 [2026-10-19 20:46:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.4030 [2026-10-19 20:46:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.1688 [2026-10-19 20:46:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.3214 [2026-10-19 20:47:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.4314 [2026-10-19 20:47:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.3747 [2026-10-19 20:47:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.6037 [2026-10-19 20:47:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.3628 [2026-10-19 20:47:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.4464 [2026-10-19 20:47:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.4375 [2026-10-19 20:47:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.5360 [2026-10-19 20:47:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.2767 [2026-10-19 20:47:37][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,570 | MFU: 1.8% [2026-10-19 20:47:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.4410 [2026-10-19 20:47:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.5158 [2026-10-19 20:47:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.9152 [2026-10-19 20:47:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.6780 [2026-10-19 20:47:42][INFO][training.pretrain_engine] [Ph2|Step 53] loss=10.4449 | ppl=34369.18 | aux=0.02720 | lr=6.40e-05 | gnorm=1.570 | tok/s=29,273 | MFU=7.0% | step=143282ms | tokens=0.063B | mem=4.2/32.7GB | cost=$27.83/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 18:32 (141.7h) | Phase3: SKIPPED | SFT: Oct 25 23:16 (146.5h) | DPO (Final): Oct 25 23:50 (147.1h) [2026-10-19 20:47:42][INFO][training.pretrain_engine] Starting global step 54 (accumulating 32 micro-batches)... [2026-10-19 20:47:42][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 20:47:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.2476 [2026-10-19 20:47:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.2150 [2026-10-19 20:47:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.1581 [2026-10-19 20:47:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.1968 [2026-10-19 20:48:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.1386 [2026-10-19 20:48:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.0982 [2026-10-19 20:48:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.3971 [2026-10-19 20:48:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.1789 [2026-10-19 20:48:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.4046 [2026-10-19 20:48:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.7481 [2026-10-19 20:48:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.4898 [2026-10-19 20:48:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.5078 [2026-10-19 20:48:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.4441 [2026-10-19 20:48:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.6349 [2026-10-19 20:48:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.3734 [2026-10-19 20:48:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.7540 [2026-10-19 20:49:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.4858 [2026-10-19 20:49:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.3643 [2026-10-19 20:49:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.3005 [2026-10-19 20:49:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.7695 [2026-10-19 20:49:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.5673 [2026-10-19 20:49:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.2565 [2026-10-19 20:49:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.6521 [2026-10-19 20:49:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.3796 [2026-10-19 20:49:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.2294 [2026-10-19 20:49:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.3748 [2026-10-19 20:49:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.4299 [2026-10-19 20:49:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.4369 [2026-10-19 20:50:00][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,563 | MFU: 1.8% [2026-10-19 20:50:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.3845 [2026-10-19 20:50:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.3871 [2026-10-19 20:50:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.4017 [2026-10-19 20:50:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.3372 [2026-10-19 20:50:05][INFO][training.pretrain_engine] [Ph2|Step 54] loss=10.3887 | ppl=32489.04 | aux=0.02704 | lr=6.45e-05 | gnorm=1.508 | tok/s=29,274 | MFU=7.0% | step=143280ms | tokens=0.064B | mem=4.2/32.7GB | cost=$28.35/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 18:33 (141.7h) | Phase3: SKIPPED | SFT: Oct 25 23:18 (146.5h) | DPO (Final): Oct 25 23:52 (147.0h) [2026-10-19 20:50:08][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step54 | Tokens: 0.064B | Spent: $28.36 | Remaining: $371.64 [2026-10-19 20:50:08][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step54 [2026-10-19 20:50:08][INFO][training.pretrain_engine] Starting global step 55 (accumulating 32 micro-batches)... [2026-10-19 20:50:08][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 20:50:08][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-19 20:50:08][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-19 20:50:08][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-19 20:50:08][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step54 to Phase-Technologies/forge-500m-pretrain (path: phase2_step54)... [2026-10-19 20:50:08][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-19 20:50:17][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-19 20:50:17][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step54 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-19 20:50:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.2151 [2026-10-19 20:50:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.2954 [2026-10-19 20:50:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.1945 [2026-10-19 20:50:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.4609 [2026-10-19 20:50:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.5581 [2026-10-19 20:50:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.0497 [2026-10-19 20:50:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.6512 [2026-10-19 20:50:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.9931 [2026-10-19 20:50:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.2139 [2026-10-19 20:50:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.5199 [2026-10-19 20:50:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.2752 [2026-10-19 20:51:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.3493 [2026-10-19 20:51:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.2804 [2026-10-19 20:51:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.3401 [2026-10-19 20:51:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.3020 [2026-10-19 20:51:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.2635 [2026-10-19 20:51:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.6367 [2026-10-19 20:51:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.2595 [2026-10-19 20:51:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.1479 [2026-10-19 20:51:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.5355 [2026-10-19 20:51:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.2312 [2026-10-19 20:51:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.2067 [2026-10-19 20:51:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.5241 [2026-10-19 20:51:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.5147 [2026-10-19 20:52:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.4820 [2026-10-19 20:52:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.7885 [2026-10-19 20:52:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.7773 [2026-10-19 20:52:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.8431 [2026-10-19 20:52:27][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,536 | MFU: 1.8% [2026-10-19 20:52:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.4314 [2026-10-19 20:52:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.3847 [2026-10-19 20:52:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.3019 [2026-10-19 20:52:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.1502 [2026-10-19 20:52:31][INFO][training.pretrain_engine] [Ph2|Step 55] loss=10.4546 | ppl=34705.13 | aux=0.02686 | lr=6.51e-05 | gnorm=1.461 | tok/s=29,272 | MFU=7.0% | step=143287ms | tokens=0.065B | mem=4.2/32.7GB | cost=$28.89/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 18:36 (141.7h) | Phase3: SKIPPED | SFT: Oct 25 23:20 (146.5h) | DPO (Final): Oct 25 23:54 (147.0h) [2026-10-19 20:52:31][INFO][training.pretrain_engine] Starting global step 56 (accumulating 32 micro-batches)... [2026-10-19 20:52:31][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 20:52:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.2280 [2026-10-19 20:52:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.2087 [2026-10-19 20:52:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.2811 [2026-10-19 20:52:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.2040 [2026-10-19 20:53:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.3029 [2026-10-19 20:53:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.2804 [2026-10-19 20:53:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.1327 [2026-10-19 20:53:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.5616 [2026-10-19 20:53:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.3488 [2026-10-19 20:53:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.1691 [2026-10-19 20:53:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.3997 [2026-10-19 20:53:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.1236 [2026-10-19 20:53:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.4708 [2026-10-19 20:53:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.3230 [2026-10-19 20:53:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.5134 [2026-10-19 20:53:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.3976 [2026-10-19 20:54:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.5312 [2026-10-19 20:54:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.1110 [2026-10-19 20:54:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.5925 [2026-10-19 20:54:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.7095 [2026-10-19 20:54:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.0778 [2026-10-19 20:54:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.0028 [2026-10-19 20:54:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.1101 [2026-10-19 20:54:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.9838 [2026-10-19 20:54:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.3351 [2026-10-19 20:54:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.0405 [2026-10-19 20:54:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.4335 [2026-10-19 20:54:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.1491 [2026-10-19 20:54:50][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,552 | MFU: 1.8% [2026-10-19 20:54:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.7436 [2026-10-19 20:54:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.7397 [2026-10-19 20:54:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.5641 [2026-10-19 20:54:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.4789 [2026-10-19 20:54:55][INFO][training.pretrain_engine] [Ph2|Step 56] loss=10.2931 | ppl=29528.18 | aux=0.02686 | lr=6.57e-05 | gnorm=1.609 | tok/s=29,272 | MFU=7.0% | step=143288ms | tokens=0.066B | mem=4.2/32.7GB | cost=$29.41/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 18:37 (141.7h) | Phase3: SKIPPED | SFT: Oct 25 23:22 (146.5h) | DPO (Final): Oct 25 23:56 (147.0h) [2026-10-19 20:54:55][INFO][training.pretrain_engine] Starting global step 57 (accumulating 32 micro-batches)... [2026-10-19 20:54:55][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 20:55:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.0371 [2026-10-19 20:55:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.1012 [2026-10-19 20:55:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.1441 [2026-10-19 20:55:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.9791 [2026-10-19 20:55:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.3170 [2026-10-19 20:55:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.3197 [2026-10-19 20:55:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.3574 [2026-10-19 20:55:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.2865 [2026-10-19 20:55:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.5352 [2026-10-19 20:55:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.5406 [2026-10-19 20:55:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.4430 [2026-10-19 20:55:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.4208 [2026-10-19 20:56:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.1744 [2026-10-19 20:56:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.2148 [2026-10-19 20:56:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.1679 [2026-10-19 20:56:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.1874 [2026-10-19 20:56:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.3777 [2026-10-19 20:56:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.9334 [2026-10-19 20:56:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.3145 [2026-10-19 20:56:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.0897 [2026-10-19 20:56:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.3370 [2026-10-19 20:56:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.2333 [2026-10-19 20:56:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.2948 [2026-10-19 20:56:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.3893 [2026-10-19 20:56:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.2671 [2026-10-19 20:56:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.1893 [2026-10-19 20:57:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.0530 [2026-10-19 20:57:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.1516 [2026-10-19 20:57:14][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,546 | MFU: 1.8% [2026-10-19 20:57:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.1821 [2026-10-19 20:57:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.2774 [2026-10-19 20:57:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.3941 [2026-10-19 20:57:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.3210 [2026-10-19 20:57:18][INFO][training.pretrain_engine] [Ph2|Step 57] loss=10.3359 | ppl=30820.73 | aux=0.02672 | lr=6.62e-05 | gnorm=1.539 | tok/s=29,271 | MFU=7.0% | step=143292ms | tokens=0.067B | mem=4.2/32.7GB | cost=$29.93/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 18:39 (141.7h) | Phase3: SKIPPED | SFT: Oct 25 23:24 (146.5h) | DPO (Final): Oct 25 23:58 (147.0h) [2026-10-19 20:57:18][INFO][training.pretrain_engine] Starting global step 58 (accumulating 32 micro-batches)... [2026-10-19 20:57:18][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 20:57:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.7725 [2026-10-19 20:57:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.4168 [2026-10-19 20:57:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.3777 [2026-10-19 20:57:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.9174 [2026-10-19 20:57:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.9988 [2026-10-19 20:57:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.1992 [2026-10-19 20:57:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.1053 [2026-10-19 20:57:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.9899 [2026-10-19 20:58:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.2581 [2026-10-19 20:58:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.2380 [2026-10-19 20:58:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.0033 [2026-10-19 20:58:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.3144 [2026-10-19 20:58:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.3490 [2026-10-19 20:58:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.2030 [2026-10-19 20:58:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.2154 [2026-10-19 20:58:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.3303 [2026-10-19 20:58:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.8810 [2026-10-19 20:58:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.7732 [2026-10-19 20:58:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.4274 [2026-10-19 20:58:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.7783 [2026-10-19 20:59:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.5024 [2026-10-19 20:59:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.9342 [2026-10-19 20:59:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.3443 [2026-10-19 20:59:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.9076 [2026-10-19 20:59:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.2789 [2026-10-19 20:59:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.2693 [2026-10-19 20:59:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.3698 [2026-10-19 20:59:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.0935 [2026-10-19 20:59:37][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,557 | MFU: 1.8% [2026-10-19 20:59:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.5567 [2026-10-19 20:59:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.3034 [2026-10-19 20:59:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.3732 [2026-10-19 20:59:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.4921 [2026-10-19 20:59:42][INFO][training.pretrain_engine] [Ph2|Step 58] loss=10.3522 | ppl=31324.91 | aux=0.02671 | lr=6.68e-05 | gnorm=1.461 | tok/s=29,271 | MFU=7.0% | step=143291ms | tokens=0.068B | mem=4.2/32.7GB | cost=$30.46/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 18:41 (141.7h) | Phase3: SKIPPED | SFT: Oct 25 23:26 (146.4h) | DPO (Final): Oct 26 00:00 (147.0h) [2026-10-19 20:59:42][INFO][training.pretrain_engine] Starting global step 59 (accumulating 32 micro-batches)... [2026-10-19 20:59:42][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 20:59:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.2734 [2026-10-19 20:59:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.4487 [2026-10-19 21:00:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.0799 [2026-10-19 21:00:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.6518 [2026-10-19 21:00:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.2844 [2026-10-19 21:00:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.2799 [2026-10-19 21:00:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.3091 [2026-10-19 21:00:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.7608 [2026-10-19 21:00:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.9448 [2026-10-19 21:00:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.0495 [2026-10-19 21:00:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.9893 [2026-10-19 21:00:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.0657 [2026-10-19 21:00:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=11.0100 [2026-10-19 21:00:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.3242 [2026-10-19 21:00:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.9942 [2026-10-19 21:00:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.2278 [2026-10-19 21:01:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.1129 [2026-10-19 21:01:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.9368 [2026-10-19 21:01:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.9477 [2026-10-19 21:01:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.0665 [2026-10-19 21:01:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.1071 [2026-10-19 21:01:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.4326 [2026-10-19 21:01:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.9618 [2026-10-19 21:01:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.8818 [2026-10-19 21:01:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.4182 [2026-10-19 21:01:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.1918 [2026-10-19 21:01:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.3228 [2026-10-19 21:01:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.4807 [2026-10-19 21:02:00][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,553 | MFU: 1.8% [2026-10-19 21:02:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.2411 [2026-10-19 21:02:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.5084 [2026-10-19 21:02:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.6487 [2026-10-19 21:02:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.2781 [2026-10-19 21:02:05][INFO][training.pretrain_engine] [Ph2|Step 59] loss=10.2962 | ppl=29620.16 | aux=0.02661 | lr=6.74e-05 | gnorm=1.531 | tok/s=29,271 | MFU=7.0% | step=143292ms | tokens=0.069B | mem=4.2/32.7GB | cost=$30.98/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 18:43 (141.7h) | Phase3: SKIPPED | SFT: Oct 25 23:28 (146.4h) | DPO (Final): Oct 26 00:02 (147.0h) [2026-10-19 21:02:05][INFO][training.pretrain_engine] Starting global step 60 (accumulating 32 micro-batches)... [2026-10-19 21:02:05][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 21:02:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.9648 [2026-10-19 21:02:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.9302 [2026-10-19 21:02:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.9989 [2026-10-19 21:02:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.8162 [2026-10-19 21:02:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.0801 [2026-10-19 21:02:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.1211 [2026-10-19 21:02:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.2197 [2026-10-19 21:02:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.1515 [2026-10-19 21:02:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.9371 [2026-10-19 21:02:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.1055 [2026-10-19 21:02:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.1533 [2026-10-19 21:02:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.0633 [2026-10-19 21:03:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.9667 [2026-10-19 21:03:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.2313 [2026-10-19 21:03:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.2916 [2026-10-19 21:03:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.0612 [2026-10-19 21:03:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.2127 [2026-10-19 21:03:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.0051 [2026-10-19 21:03:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.2802 [2026-10-19 21:03:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.0598 [2026-10-19 21:03:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.4537 [2026-10-19 21:03:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.8946 [2026-10-19 21:03:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.2642 [2026-10-19 21:03:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.1449 [2026-10-19 21:04:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.0854 [2026-10-19 21:04:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.2279 [2026-10-19 21:04:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.0845 [2026-10-19 21:04:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.1571 [2026-10-19 21:04:23][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,586 | MFU: 1.8% [2026-10-19 21:04:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.4776 [2026-10-19 21:04:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.1389 [2026-10-19 21:04:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.3943 [2026-10-19 21:04:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.5560 [2026-10-19 21:04:28][INFO][training.pretrain_engine] [Ph2|Step 60] loss=10.2516 | ppl=28327.92 | aux=0.02655 | lr=6.79e-05 | gnorm=1.453 | tok/s=29,271 | MFU=7.0% | step=143291ms | tokens=0.070B | mem=4.2/32.7GB | cost=$31.51/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 18:45 (141.7h) | Phase3: SKIPPED | SFT: Oct 25 23:29 (146.4h) | DPO (Final): Oct 26 00:04 (147.0h) [2026-10-19 21:04:28][INFO][training.pretrain_engine] Starting global step 61 (accumulating 32 micro-batches)... [2026-10-19 21:04:28][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 21:04:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.9331 [2026-10-19 21:04:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.9356 [2026-10-19 21:04:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.0364 [2026-10-19 21:04:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.1428 [2026-10-19 21:05:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.0575 [2026-10-19 21:05:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.1504 [2026-10-19 21:05:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.1442 [2026-10-19 21:05:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.1070 [2026-10-19 21:05:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.7924 [2026-10-19 21:05:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.4891 [2026-10-19 21:05:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.9616 [2026-10-19 21:05:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.0931 [2026-10-19 21:05:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.3806 [2026-10-19 21:05:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.0684 [2026-10-19 21:05:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.4141 [2026-10-19 21:05:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.4225 [2026-10-19 21:05:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.9955 [2026-10-19 21:05:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.3026 [2026-10-19 21:05:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.1633 [2026-10-19 21:05:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.1723 [2026-10-19 21:06:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.0094 [2026-10-19 21:06:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.0282 [2026-10-19 21:06:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.2630 [2026-10-19 21:06:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.4485 [2026-10-19 21:06:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.1769 [2026-10-19 21:06:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.1889 [2026-10-19 21:06:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.1865 [2026-10-19 21:06:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.2654 [2026-10-19 21:06:47][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,574 | MFU: 1.8% [2026-10-19 21:06:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.1302 [2026-10-19 21:06:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.0155 [2026-10-19 21:06:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.0685 [2026-10-19 21:06:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.8634 [2026-10-19 21:06:51][INFO][training.pretrain_engine] [Ph2|Step 61] loss=10.1411 | ppl=25364.62 | aux=0.02659 | lr=6.85e-05 | gnorm=1.461 | tok/s=29,272 | MFU=7.0% | step=143288ms | tokens=0.071B | mem=4.2/32.7GB | cost=$32.03/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 18:46 (141.7h) | Phase3: SKIPPED | SFT: Oct 25 23:31 (146.4h) | DPO (Final): Oct 26 00:05 (147.0h) [2026-10-19 21:06:51][INFO][training.pretrain_engine] Starting global step 62 (accumulating 32 micro-batches)... [2026-10-19 21:06:51][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 21:07:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.8166 [2026-10-19 21:07:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.8182 [2026-10-19 21:07:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.9907 [2026-10-19 21:07:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.8620 [2026-10-19 21:07:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.8255 [2026-10-19 21:07:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.9124 [2026-10-19 21:07:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.9292 [2026-10-19 21:07:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.9910 [2026-10-19 21:07:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.0323 [2026-10-19 21:07:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.0925 [2026-10-19 21:07:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.1727 [2026-10-19 21:07:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.9829 [2026-10-19 21:08:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.1851 [2026-10-19 21:08:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.9030 [2026-10-19 21:08:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.0881 [2026-10-19 21:08:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.1397 [2026-10-19 21:08:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.7665 [2026-10-19 21:08:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.9051 [2026-10-19 21:08:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.8019 [2026-10-19 21:08:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.6851 [2026-10-19 21:08:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.0241 [2026-10-19 21:08:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.0958 [2026-10-19 21:08:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.2296 [2026-10-19 21:08:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.9390 [2026-10-19 21:08:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.4080 [2026-10-19 21:08:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.3641 [2026-10-19 21:08:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.3193 [2026-10-19 21:08:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.9284 [2026-10-19 21:09:09][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,586 | MFU: 1.8% [2026-10-19 21:09:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.2380 [2026-10-19 21:09:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.0899 [2026-10-19 21:09:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.2669 [2026-10-19 21:09:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.4454 [2026-10-19 21:09:14][INFO][training.pretrain_engine] [Ph2|Step 62] loss=10.2641 | ppl=28683.79 | aux=0.02635 | lr=6.91e-05 | gnorm=1.438 | tok/s=29,273 | MFU=7.0% | step=143281ms | tokens=0.072B | mem=4.2/32.7GB | cost=$32.55/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 18:48 (141.7h) | Phase3: SKIPPED | SFT: Oct 25 23:32 (146.4h) | DPO (Final): Oct 26 00:07 (147.0h) [2026-10-19 21:09:14][INFO][training.pretrain_engine] Starting global step 63 (accumulating 32 micro-batches)... [2026-10-19 21:09:14][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 21:09:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.9302 [2026-10-19 21:09:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.0150 [2026-10-19 21:09:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.9481 [2026-10-19 21:09:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.0421 [2026-10-19 21:09:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.2627 [2026-10-19 21:09:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=11.0772 [2026-10-19 21:09:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.2375 [2026-10-19 21:09:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.1134 [2026-10-19 21:10:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.1237 [2026-10-19 21:10:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.1915 [2026-10-19 21:10:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.5612 [2026-10-19 21:10:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.5320 [2026-10-19 21:10:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.2291 [2026-10-19 21:10:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.2802 [2026-10-19 21:10:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.4582 [2026-10-19 21:10:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.5389 [2026-10-19 21:10:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.1118 [2026-10-19 21:10:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.1595 [2026-10-19 21:10:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.9746 [2026-10-19 21:10:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.1566 [2026-10-19 21:11:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.3877 [2026-10-19 21:11:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.6401 [2026-10-19 21:11:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.5316 [2026-10-19 21:11:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.3885 [2026-10-19 21:11:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.3048 [2026-10-19 21:11:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.9364 [2026-10-19 21:11:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.8790 [2026-10-19 21:11:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.0730 [2026-10-19 21:11:32][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,591 | MFU: 1.8% [2026-10-19 21:11:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.0595 [2026-10-19 21:11:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.3555 [2026-10-19 21:11:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.9169 [2026-10-19 21:11:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.1576 [2026-10-19 21:11:37][INFO][training.pretrain_engine] [Ph2|Step 63] loss=10.2606 | ppl=28583.38 | aux=0.02639 | lr=6.96e-05 | gnorm=1.344 | tok/s=29,274 | MFU=7.0% | step=143278ms | tokens=0.073B | mem=4.2/32.7GB | cost=$33.07/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 18:49 (141.6h) | Phase3: SKIPPED | SFT: Oct 25 23:34 (146.4h) | DPO (Final): Oct 26 00:08 (147.0h) [2026-10-19 21:11:40][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step63 | Tokens: 0.073B | Spent: $33.08 | Remaining: $366.92 [2026-10-19 21:11:40][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step63 [2026-10-19 21:11:40][INFO][training.pretrain_engine] Starting global step 64 (accumulating 32 micro-batches)... [2026-10-19 21:11:40][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 21:11:40][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-19 21:11:40][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-19 21:11:40][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-19 21:11:40][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step63 to Phase-Technologies/forge-500m-pretrain (path: phase2_step63)... [2026-10-19 21:11:40][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-19 21:11:50][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-19 21:11:50][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step63 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-19 21:11:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.8482 [2026-10-19 21:11:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.1806 [2026-10-19 21:11:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.4613 [2026-10-19 21:11:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.9004 [2026-10-19 21:12:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.9692 [2026-10-19 21:12:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.8425 [2026-10-19 21:12:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.9263 [2026-10-19 21:12:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.9395 [2026-10-19 21:12:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.9059 [2026-10-19 21:12:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.1626 [2026-10-19 21:12:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.7129 [2026-10-19 21:12:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.8483 [2026-10-19 21:12:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.0367 [2026-10-19 21:12:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.1679 [2026-10-19 21:12:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.8819 [2026-10-19 21:12:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.9528 [2026-10-19 21:13:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.9748 [2026-10-19 21:13:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.7611 [2026-10-19 21:13:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.9759 [2026-10-19 21:13:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.0600 [2026-10-19 21:13:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.3396 [2026-10-19 21:13:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.0832 [2026-10-19 21:13:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.9212 [2026-10-19 21:13:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.0867 [2026-10-19 21:13:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.2781 [2026-10-19 21:13:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.8789 [2026-10-19 21:13:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.8033 [2026-10-19 21:13:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.8515 [2026-10-19 21:13:58][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,601 | MFU: 1.8% [2026-10-19 21:14:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.2449 [2026-10-19 21:14:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.5016 [2026-10-19 21:14:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.2790 [2026-10-19 21:14:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.4063 [2026-10-19 21:14:03][INFO][training.pretrain_engine] [Ph2|Step 64] loss=10.1400 | ppl=25337.54 | aux=0.02631 | lr=7.02e-05 | gnorm=1.406 | tok/s=29,276 | MFU=7.0% | step=143266ms | tokens=0.074B | mem=4.2/32.7GB | cost=$33.61/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 18:50 (141.6h) | Phase3: SKIPPED | SFT: Oct 25 23:35 (146.4h) | DPO (Final): Oct 26 00:09 (146.9h) [2026-10-19 21:14:03][INFO][training.pretrain_engine] Starting global step 65 (accumulating 32 micro-batches)... [2026-10-19 21:14:03][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 21:14:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.2677 [2026-10-19 21:14:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.2320 [2026-10-19 21:14:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.1925 [2026-10-19 21:14:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.3798 [2026-10-19 21:14:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.8915 [2026-10-19 21:14:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.0463 [2026-10-19 21:14:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.0975 [2026-10-19 21:14:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.1231 [2026-10-19 21:14:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.1070 [2026-10-19 21:14:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.1349 [2026-10-19 21:14:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.8951 [2026-10-19 21:14:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.1793 [2026-10-19 21:15:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.7777 [2026-10-19 21:15:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.6651 [2026-10-19 21:15:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.6557 [2026-10-19 21:15:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.0455 [2026-10-19 21:15:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.0990 [2026-10-19 21:15:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.9059 [2026-10-19 21:15:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.1111 [2026-10-19 21:15:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.0272 [2026-10-19 21:15:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.4481 [2026-10-19 21:15:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.3567 [2026-10-19 21:15:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.3108 [2026-10-19 21:15:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.2347 [2026-10-19 21:16:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.3650 [2026-10-19 21:16:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.3048 [2026-10-19 21:16:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.8828 [2026-10-19 21:16:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.8874 [2026-10-19 21:16:22][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,534 | MFU: 1.8% [2026-10-19 21:16:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.4385 [2026-10-19 21:16:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.2064 [2026-10-19 21:16:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.9662 [2026-10-19 21:16:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.0130 [2026-10-19 21:16:26][INFO][training.pretrain_engine] [Ph2|Step 65] loss=10.0493 | ppl=23138.56 | aux=0.02626 | lr=7.08e-05 | gnorm=1.445 | tok/s=29,275 | MFU=7.0% | step=143273ms | tokens=0.075B | mem=4.2/32.7GB | cost=$34.13/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 18:53 (141.6h) | Phase3: SKIPPED | SFT: Oct 25 23:37 (146.4h) | DPO (Final): Oct 26 00:12 (146.9h) [2026-10-19 21:16:26][INFO][training.pretrain_engine] Starting global step 66 (accumulating 32 micro-batches)... [2026-10-19 21:16:26][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 21:16:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.9732 [2026-10-19 21:16:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.9054 [2026-10-19 21:16:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.8008 [2026-10-19 21:16:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.7617 [2026-10-19 21:17:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.2700 [2026-10-19 21:17:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.2059 [2026-10-19 21:17:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.6320 [2026-10-19 21:17:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.3226 [2026-10-19 21:17:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.7231 [2026-10-19 21:17:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.2279 [2026-10-19 21:17:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.0895 [2026-10-19 21:17:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.1384 [2026-10-19 21:17:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.1864 [2026-10-19 21:17:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.3156 [2026-10-19 21:17:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.2077 [2026-10-19 21:17:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.1925 [2026-10-19 21:17:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.8380 [2026-10-19 21:17:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.0786 [2026-10-19 21:17:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.8249 [2026-10-19 21:17:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.9683 [2026-10-19 21:18:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.6816 [2026-10-19 21:18:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.5014 [2026-10-19 21:18:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.6213 [2026-10-19 21:18:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.5876 [2026-10-19 21:18:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.9744 [2026-10-19 21:18:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.8327 [2026-10-19 21:18:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.1055 [2026-10-19 21:18:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.8737 [2026-10-19 21:18:46][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,520 | MFU: 1.8% [2026-10-19 21:18:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.3302 [2026-10-19 21:18:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.1824 [2026-10-19 21:18:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.4731 [2026-10-19 21:18:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.3891 [2026-10-19 21:18:51][INFO][training.pretrain_engine] [Ph2|Step 66] loss=10.1191 | ppl=24812.40 | aux=0.02621 | lr=7.13e-05 | gnorm=1.359 | tok/s=29,272 | MFU=7.0% | step=143287ms | tokens=0.077B | mem=4.2/32.7GB | cost=$34.66/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 18:55 (141.6h) | Phase3: SKIPPED | SFT: Oct 25 23:40 (146.4h) | DPO (Final): Oct 26 00:14 (146.9h) [2026-10-19 21:18:51][INFO][training.pretrain_engine] Starting global step 67 (accumulating 32 micro-batches)... [2026-10-19 21:18:51][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 21:19:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.5657 [2026-10-19 21:19:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.2269 [2026-10-19 21:19:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.3050 [2026-10-19 21:19:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.4157 [2026-10-19 21:19:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.7438 [2026-10-19 21:19:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.7687 [2026-10-19 21:19:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.7457 [2026-10-19 21:19:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.6833 [2026-10-19 21:19:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.0877 [2026-10-19 21:19:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.0359 [2026-10-19 21:19:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.8689 [2026-10-19 21:19:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.2146 [2026-10-19 21:20:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.8967 [2026-10-19 21:20:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.7736 [2026-10-19 21:20:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.8432 [2026-10-19 21:20:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.9428 [2026-10-19 21:20:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.7817 [2026-10-19 21:20:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.8955 [2026-10-19 21:20:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.8712 [2026-10-19 21:20:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.8949 [2026-10-19 21:20:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.6448 [2026-10-19 21:20:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.8281 [2026-10-19 21:20:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.5889 [2026-10-19 21:20:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.5817 [2026-10-19 21:20:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.8609 [2026-10-19 21:20:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.9215 [2026-10-19 21:20:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.0286 [2026-10-19 21:20:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.9085 [2026-10-19 21:21:10][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,547 | MFU: 1.8% [2026-10-19 21:21:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.9893 [2026-10-19 21:21:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.9810 [2026-10-19 21:21:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.9444 [2026-10-19 21:21:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.1195 [2026-10-19 21:21:14][INFO][training.pretrain_engine] [Ph2|Step 67] loss=10.0585 | ppl=23353.73 | aux=0.02618 | lr=7.19e-05 | gnorm=1.430 | tok/s=29,272 | MFU=7.0% | step=143289ms | tokens=0.078B | mem=4.2/32.7GB | cost=$35.18/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 18:57 (141.6h) | Phase3: SKIPPED | SFT: Oct 25 23:42 (146.4h) | DPO (Final): Oct 26 00:16 (146.9h) [2026-10-19 21:21:14][INFO][training.pretrain_engine] Starting global step 68 (accumulating 32 micro-batches)... [2026-10-19 21:21:14][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 21:21:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.0273 [2026-10-19 21:21:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.0403 [2026-10-19 21:21:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.7350 [2026-10-19 21:21:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.0677 [2026-10-19 21:21:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.9958 [2026-10-19 21:21:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.8289 [2026-10-19 21:21:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.8800 [2026-10-19 21:21:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.7508 [2026-10-19 21:22:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.9897 [2026-10-19 21:22:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.8416 [2026-10-19 21:22:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.1552 [2026-10-19 21:22:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.7612 [2026-10-19 21:22:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.5429 [2026-10-19 21:22:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.5457 [2026-10-19 21:22:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.6123 [2026-10-19 21:22:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.4431 [2026-10-19 21:22:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.3079 [2026-10-19 21:22:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.1472 [2026-10-19 21:22:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.1737 [2026-10-19 21:22:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.5562 [2026-10-19 21:23:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.1291 [2026-10-19 21:23:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.7537 [2026-10-19 21:23:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.2012 [2026-10-19 21:23:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.0416 [2026-10-19 21:23:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.5447 [2026-10-19 21:23:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.3519 [2026-10-19 21:23:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.8941 [2026-10-19 21:23:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.0086 [2026-10-19 21:23:32][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,576 | MFU: 1.8% [2026-10-19 21:23:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.2910 [2026-10-19 21:23:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.3358 [2026-10-19 21:23:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.3153 [2026-10-19 21:23:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.6467 [2026-10-19 21:23:37][INFO][training.pretrain_engine] [Ph2|Step 68] loss=10.0719 | ppl=23669.56 | aux=0.02603 | lr=7.25e-05 | gnorm=1.406 | tok/s=29,272 | MFU=7.0% | step=143286ms | tokens=0.079B | mem=4.2/32.7GB | cost=$35.71/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 18:59 (141.6h) | Phase3: SKIPPED | SFT: Oct 25 23:44 (146.3h) | DPO (Final): Oct 26 00:18 (146.9h) [2026-10-19 21:23:37][INFO][training.pretrain_engine] Starting global step 69 (accumulating 32 micro-batches)... [2026-10-19 21:23:37][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 21:23:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.4236 [2026-10-19 21:23:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.4544 [2026-10-19 21:23:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.3178 [2026-10-19 21:23:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.0754 [2026-10-19 21:24:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.6325 [2026-10-19 21:24:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.7395 [2026-10-19 21:24:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.7142 [2026-10-19 21:24:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.7287 [2026-10-19 21:24:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.3102 [2026-10-19 21:24:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.1776 [2026-10-19 21:24:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.0768 [2026-10-19 21:24:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.1296 [2026-10-19 21:24:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.6641 [2026-10-19 21:24:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.8561 [2026-10-19 21:24:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.7614 [2026-10-19 21:24:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.6106 [2026-10-19 21:25:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.9922 [2026-10-19 21:25:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.9334 [2026-10-19 21:25:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.9660 [2026-10-19 21:25:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.1255 [2026-10-19 21:25:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.9615 [2026-10-19 21:25:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.8611 [2026-10-19 21:25:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.8945 [2026-10-19 21:25:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.9335 [2026-10-19 21:25:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.5842 [2026-10-19 21:25:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.7622 [2026-10-19 21:25:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.8479 [2026-10-19 21:25:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.5814 [2026-10-19 21:25:56][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,551 | MFU: 1.8% [2026-10-19 21:26:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.1504 [2026-10-19 21:26:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.4345 [2026-10-19 21:26:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.1400 [2026-10-19 21:26:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.1682 [2026-10-19 21:26:01][INFO][training.pretrain_engine] [Ph2|Step 69] loss=9.9959 | ppl=21936.38 | aux=0.02604 | lr=7.30e-05 | gnorm=1.320 | tok/s=29,272 | MFU=7.0% | step=143288ms | tokens=0.080B | mem=4.2/32.7GB | cost=$36.23/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 19:01 (141.6h) | Phase3: SKIPPED | SFT: Oct 25 23:45 (146.3h) | DPO (Final): Oct 26 00:20 (146.9h) [2026-10-19 21:26:01][INFO][training.pretrain_engine] Starting global step 70 (accumulating 32 micro-batches)... [2026-10-19 21:26:01][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 21:26:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.7117 [2026-10-19 21:26:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.9038 [2026-10-19 21:26:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.6176 [2026-10-19 21:26:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.0561 [2026-10-19 21:26:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.4954 [2026-10-19 21:26:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.6593 [2026-10-19 21:26:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.3933 [2026-10-19 21:26:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.6584 [2026-10-19 21:26:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.3318 [2026-10-19 21:26:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.8903 [2026-10-19 21:26:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.0180 [2026-10-19 21:26:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.0738 [2026-10-19 21:27:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.0433 [2026-10-19 21:27:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.0180 [2026-10-19 21:27:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.9614 [2026-10-19 21:27:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.9839 [2026-10-19 21:27:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.7134 [2026-10-19 21:27:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.6869 [2026-10-19 21:27:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.8483 [2026-10-19 21:27:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.7056 [2026-10-19 21:27:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.1434 [2026-10-19 21:27:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.0100 [2026-10-19 21:27:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.0592 [2026-10-19 21:27:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.1044 [2026-10-19 21:28:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.9226 [2026-10-19 21:28:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.3158 [2026-10-19 21:28:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.2938 [2026-10-19 21:28:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.1954 [2026-10-19 21:28:19][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,545 | MFU: 1.8% [2026-10-19 21:28:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.8928 [2026-10-19 21:28:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.7623 [2026-10-19 21:28:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.7930 [2026-10-19 21:28:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.3676 [2026-10-19 21:28:24][INFO][training.pretrain_engine] [Ph2|Step 70] loss=9.9397 | ppl=20737.63 | aux=0.02606 | lr=7.36e-05 | gnorm=1.344 | tok/s=29,271 | MFU=7.0% | step=143291ms | tokens=0.081B | mem=4.2/32.7GB | cost=$36.75/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 19:03 (141.6h) | Phase3: SKIPPED | SFT: Oct 25 23:47 (146.3h) | DPO (Final): Oct 26 00:22 (146.9h) [2026-10-19 21:28:24][INFO][training.pretrain_engine] Starting global step 71 (accumulating 32 micro-batches)... [2026-10-19 21:28:24][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 21:28:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.1545 [2026-10-19 21:28:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.8603 [2026-10-19 21:28:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.0499 [2026-10-19 21:28:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.7614 [2026-10-19 21:29:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.5739 [2026-10-19 21:29:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.7174 [2026-10-19 21:29:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.5565 [2026-10-19 21:29:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.0560 [2026-10-19 21:29:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.5975 [2026-10-19 21:29:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.5435 [2026-10-19 21:29:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.5713 [2026-10-19 21:29:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.5315 [2026-10-19 21:29:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.5638 [2026-10-19 21:29:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.5354 [2026-10-19 21:29:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.6144 [2026-10-19 21:29:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.8640 [2026-10-19 21:29:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.0681 [2026-10-19 21:29:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.8064 [2026-10-19 21:29:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.8688 [2026-10-19 21:29:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.0172 [2026-10-19 21:30:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.0888 [2026-10-19 21:30:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.8825 [2026-10-19 21:30:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.2328 [2026-10-19 21:30:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.0311 [2026-10-19 21:30:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.8902 [2026-10-19 21:30:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.9764 [2026-10-19 21:30:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.8630 [2026-10-19 21:30:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.0910 [2026-10-19 21:30:43][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,555 | MFU: 1.8% [2026-10-19 21:30:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.8418 [2026-10-19 21:30:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.9615 [2026-10-19 21:30:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.9687 [2026-10-19 21:30:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.0517 [2026-10-19 21:30:47][INFO][training.pretrain_engine] [Ph2|Step 71] loss=9.8598 | ppl=19145.24 | aux=0.02579 | lr=7.42e-05 | gnorm=1.430 | tok/s=29,271 | MFU=7.0% | step=143291ms | tokens=0.082B | mem=4.2/32.7GB | cost=$37.28/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 19:05 (141.6h) | Phase3: SKIPPED | SFT: Oct 25 23:49 (146.3h) | DPO (Final): Oct 26 00:23 (146.9h) [2026-10-19 21:30:47][INFO][training.pretrain_engine] Starting global step 72 (accumulating 32 micro-batches)... [2026-10-19 21:30:47][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 21:31:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.6890 [2026-10-19 21:31:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.8019 [2026-10-19 21:31:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.8083 [2026-10-19 21:31:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.0508 [2026-10-19 21:31:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.2087 [2026-10-19 21:31:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.0691 [2026-10-19 21:31:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.5487 [2026-10-19 21:31:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.6670 [2026-10-19 21:31:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.6817 [2026-10-19 21:31:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.7023 [2026-10-19 21:31:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.8274 [2026-10-19 21:31:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.1057 [2026-10-19 21:31:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.7864 [2026-10-19 21:31:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.9181 [2026-10-19 21:31:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.6994 [2026-10-19 21:31:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.9439 [2026-10-19 21:32:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.4851 [2026-10-19 21:32:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.4549 [2026-10-19 21:32:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.8665 [2026-10-19 21:32:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.4605 [2026-10-19 21:32:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.8826 [2026-10-19 21:32:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.9795 [2026-10-19 21:32:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.8073 [2026-10-19 21:32:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.0262 [2026-10-19 21:32:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.7498 [2026-10-19 21:32:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.8022 [2026-10-19 21:32:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.5685 [2026-10-19 21:32:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.6501 [2026-10-19 21:33:06][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,551 | MFU: 1.8% [2026-10-19 21:33:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.6626 [2026-10-19 21:33:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.7953 [2026-10-19 21:33:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.6535 [2026-10-19 21:33:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.8515 [2026-10-19 21:33:11][INFO][training.pretrain_engine] [Ph2|Step 72] loss=9.8575 | ppl=19101.07 | aux=0.02582 | lr=7.47e-05 | gnorm=1.383 | tok/s=29,271 | MFU=7.0% | step=143292ms | tokens=0.083B | mem=4.2/32.7GB | cost=$37.80/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 19:06 (141.6h) | Phase3: SKIPPED | SFT: Oct 25 23:51 (146.3h) | DPO (Final): Oct 26 00:25 (146.9h) [2026-10-19 21:33:14][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step72 | Tokens: 0.083B | Spent: $37.81 | Remaining: $362.19 [2026-10-19 21:33:14][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step72 [2026-10-19 21:33:14][INFO][training.pretrain_engine] Starting global step 73 (accumulating 32 micro-batches)... [2026-10-19 21:33:14][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 21:33:14][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-19 21:33:14][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-19 21:33:14][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-19 21:33:14][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step72 to Phase-Technologies/forge-500m-pretrain (path: phase2_step72)... [2026-10-19 21:33:14][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-19 21:33:23][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-19 21:33:23][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step72 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-19 21:33:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.9495 [2026-10-19 21:33:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.6867 [2026-10-19 21:33:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.9116 [2026-10-19 21:33:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.8553 [2026-10-19 21:33:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.6027 [2026-10-19 21:33:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.4319 [2026-10-19 21:33:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.5695 [2026-10-19 21:33:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.6797 [2026-10-19 21:34:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.2189 [2026-10-19 21:34:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.3203 [2026-10-19 21:34:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.6843 [2026-10-19 21:34:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.7030 [2026-10-19 21:34:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.1422 [2026-10-19 21:34:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.1650 [2026-10-19 21:34:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.0548 [2026-10-19 21:34:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.9623 [2026-10-19 21:34:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.9574 [2026-10-19 21:34:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.7936 [2026-10-19 21:34:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.7689 [2026-10-19 21:34:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.6497 [2026-10-19 21:34:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.7793 [2026-10-19 21:34:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.0896 [2026-10-19 21:34:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.9956 [2026-10-19 21:35:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.3893 [2026-10-19 21:35:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.6127 [2026-10-19 21:35:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.8123 [2026-10-19 21:35:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.7612 [2026-10-19 21:35:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.5138 [2026-10-19 21:35:32][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,603 | MFU: 1.8% [2026-10-19 21:35:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.5597 [2026-10-19 21:35:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.6120 [2026-10-19 21:35:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.5431 [2026-10-19 21:35:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.7723 [2026-10-19 21:35:36][INFO][training.pretrain_engine] [Ph2|Step 73] loss=9.8113 | ppl=18238.43 | aux=0.02575 | lr=7.53e-05 | gnorm=1.375 | tok/s=29,274 | MFU=7.0% | step=143280ms | tokens=0.084B | mem=4.2/32.7GB | cost=$38.33/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 19:07 (141.5h) | Phase3: SKIPPED | SFT: Oct 25 23:52 (146.3h) | DPO (Final): Oct 26 00:26 (146.9h) [2026-10-19 21:35:36][INFO][training.pretrain_engine] Starting global step 74 (accumulating 32 micro-batches)... [2026-10-19 21:35:36][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 21:35:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.5500 [2026-10-19 21:35:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.3777 [2026-10-19 21:35:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.6378 [2026-10-19 21:35:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.7093 [2026-10-19 21:36:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.8779 [2026-10-19 21:36:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.7514 [2026-10-19 21:36:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.0853 [2026-10-19 21:36:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.9485 [2026-10-19 21:36:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.8169 [2026-10-19 21:36:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.1071 [2026-10-19 21:36:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.7536 [2026-10-19 21:36:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.6853 [2026-10-19 21:36:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.4334 [2026-10-19 21:36:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.5199 [2026-10-19 21:36:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.4749 [2026-10-19 21:36:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.5114 [2026-10-19 21:37:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.5781 [2026-10-19 21:37:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.6364 [2026-10-19 21:37:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.6503 [2026-10-19 21:37:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.6420 [2026-10-19 21:37:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.7560 [2026-10-19 21:37:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.4878 [2026-10-19 21:37:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.9195 [2026-10-19 21:37:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.6914 [2026-10-19 21:37:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.4905 [2026-10-19 21:37:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.1060 [2026-10-19 21:37:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.1995 [2026-10-19 21:37:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.1755 [2026-10-19 21:37:54][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,586 | MFU: 1.8% [2026-10-19 21:37:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.8304 [2026-10-19 21:37:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.7133 [2026-10-19 21:37:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.7774 [2026-10-19 21:37:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.9715 [2026-10-19 21:37:59][INFO][training.pretrain_engine] [Ph2|Step 74] loss=9.8423 | ppl=18812.48 | aux=0.02574 | lr=7.59e-05 | gnorm=1.398 | tok/s=29,274 | MFU=7.0% | step=143277ms | tokens=0.085B | mem=4.2/32.7GB | cost=$38.86/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 19:09 (141.5h) | Phase3: SKIPPED | SFT: Oct 25 23:54 (146.3h) | DPO (Final): Oct 26 00:28 (146.8h) [2026-10-19 21:37:59][INFO][training.pretrain_engine] Starting global step 75 (accumulating 32 micro-batches)... [2026-10-19 21:37:59][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 21:38:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.8210 [2026-10-19 21:38:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.7667 [2026-10-19 21:38:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.8327 [2026-10-19 21:38:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.2994 [2026-10-19 21:38:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.9598 [2026-10-19 21:38:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.9110 [2026-10-19 21:38:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.9248 [2026-10-19 21:38:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.8540 [2026-10-19 21:38:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.7921 [2026-10-19 21:38:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.6261 [2026-10-19 21:38:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.6685 [2026-10-19 21:38:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.7087 [2026-10-19 21:39:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.3214 [2026-10-19 21:39:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.2239 [2026-10-19 21:39:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.2589 [2026-10-19 21:39:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.1929 [2026-10-19 21:39:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.8850 [2026-10-19 21:39:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.7021 [2026-10-19 21:39:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.5983 [2026-10-19 21:39:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.7184 [2026-10-19 21:39:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.6910 [2026-10-19 21:39:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.6070 [2026-10-19 21:39:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.6515 [2026-10-19 21:39:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.5970 [2026-10-19 21:40:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.1368 [2026-10-19 21:40:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.4385 [2026-10-19 21:40:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.1073 [2026-10-19 21:40:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.7529 [2026-10-19 21:40:18][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,557 | MFU: 1.8% [2026-10-19 21:40:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.6518 [2026-10-19 21:40:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.5529 [2026-10-19 21:40:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.7167 [2026-10-19 21:40:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.7575 [2026-10-19 21:40:23][INFO][training.pretrain_engine] [Ph2|Step 75] loss=9.8283 | ppl=18550.99 | aux=0.02574 | lr=7.64e-05 | gnorm=1.391 | tok/s=29,274 | MFU=7.0% | step=143276ms | tokens=0.086B | mem=4.2/32.7GB | cost=$39.38/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 19:11 (141.5h) | Phase3: SKIPPED | SFT: Oct 25 23:56 (146.3h) | DPO (Final): Oct 26 00:30 (146.8h) [2026-10-19 21:40:23][INFO][training.pretrain_engine] Starting global step 76 (accumulating 32 micro-batches)... [2026-10-19 21:40:23][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 21:40:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.1214 [2026-10-19 21:40:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.8680 [2026-10-19 21:40:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.1307 [2026-10-19 21:40:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.0927 [2026-10-19 21:40:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.5033 [2026-10-19 21:40:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.9589 [2026-10-19 21:40:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.6776 [2026-10-19 21:40:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.5470 [2026-10-19 21:41:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.6027 [2026-10-19 21:41:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.6610 [2026-10-19 21:41:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.5638 [2026-10-19 21:41:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.7695 [2026-10-19 21:41:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.6650 [2026-10-19 21:41:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.5845 [2026-10-19 21:41:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.7959 [2026-10-19 21:41:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.8126 [2026-10-19 21:41:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.6051 [2026-10-19 21:41:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.5688 [2026-10-19 21:41:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.6321 [2026-10-19 21:41:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.5437 [2026-10-19 21:42:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.8697 [2026-10-19 21:42:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.4390 [2026-10-19 21:42:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.4954 [2026-10-19 21:42:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.7623 [2026-10-19 21:42:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.8166 [2026-10-19 21:42:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.9506 [2026-10-19 21:42:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.2982 [2026-10-19 21:42:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.1447 [2026-10-19 21:42:41][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,556 | MFU: 1.8% [2026-10-19 21:42:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.3702 [2026-10-19 21:42:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.5639 [2026-10-19 21:42:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.9178 [2026-10-19 21:42:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.4247 [2026-10-19 21:42:46][INFO][training.pretrain_engine] [Ph2|Step 76] loss=9.7750 | ppl=17589.06 | aux=0.02562 | lr=7.70e-05 | gnorm=1.547 | tok/s=29,274 | MFU=7.0% | step=143276ms | tokens=0.087B | mem=4.2/32.7GB | cost=$39.90/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 19:13 (141.5h) | Phase3: SKIPPED | SFT: Oct 25 23:57 (146.3h) | DPO (Final): Oct 26 00:31 (146.8h) [2026-10-19 21:42:46][INFO][training.pretrain_engine] Starting global step 77 (accumulating 32 micro-batches)... [2026-10-19 21:42:46][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 21:43:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.1266 [2026-10-19 21:43:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.6808 [2026-10-19 21:43:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.3722 [2026-10-19 21:43:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.7206 [2026-10-19 21:43:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.7494 [2026-10-19 21:43:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.0734 [2026-10-19 21:43:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.7805 [2026-10-19 21:43:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.8898 [2026-10-19 21:43:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.7097 [2026-10-19 21:43:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.5453 [2026-10-19 21:43:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.4395 [2026-10-19 21:43:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.2627 [2026-10-19 21:43:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.6584 [2026-10-19 21:43:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.5721 [2026-10-19 21:43:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.5496 [2026-10-19 21:43:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.4818 [2026-10-19 21:44:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.2718 [2026-10-19 21:44:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.0427 [2026-10-19 21:44:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.1479 [2026-10-19 21:44:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=10.6469 [2026-10-19 21:44:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.6071 [2026-10-19 21:44:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.6823 [2026-10-19 21:44:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.9305 [2026-10-19 21:44:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.6238 [2026-10-19 21:44:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.4150 [2026-10-19 21:44:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.4214 [2026-10-19 21:44:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.4274 [2026-10-19 21:44:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.5258 [2026-10-19 21:45:05][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,556 | MFU: 1.8% [2026-10-19 21:45:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.9549 [2026-10-19 21:45:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.0391 [2026-10-19 21:45:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.2173 [2026-10-19 21:45:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.9709 [2026-10-19 21:45:09][INFO][training.pretrain_engine] [Ph2|Step 77] loss=9.7123 | ppl=16519.02 | aux=0.02571 | lr=7.76e-05 | gnorm=1.383 | tok/s=29,274 | MFU=7.0% | step=143277ms | tokens=0.088B | mem=4.2/32.7GB | cost=$40.43/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 19:14 (141.5h) | Phase3: SKIPPED | SFT: Oct 25 23:59 (146.2h) | DPO (Final): Oct 26 00:33 (146.8h) [2026-10-19 21:45:09][INFO][training.pretrain_engine] Starting global step 78 (accumulating 32 micro-batches)... [2026-10-19 21:45:09][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 21:45:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.8297 [2026-10-19 21:45:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.9514 [2026-10-19 21:45:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.6832 [2026-10-19 21:45:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.8816 [2026-10-19 21:45:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.5223 [2026-10-19 21:45:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.7188 [2026-10-19 21:45:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.6359 [2026-10-19 21:45:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.7802 [2026-10-19 21:46:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.5943 [2026-10-19 21:46:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.4563 [2026-10-19 21:46:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.5188 [2026-10-19 21:46:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.5176 [2026-10-19 21:46:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.4052 [2026-10-19 21:46:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.3625 [2026-10-19 21:46:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.5377 [2026-10-19 21:46:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.5473 [2026-10-19 21:46:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.7307 [2026-10-19 21:46:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.7715 [2026-10-19 21:46:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.3753 [2026-10-19 21:46:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.7413 [2026-10-19 21:46:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.4256 [2026-10-19 21:46:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.6315 [2026-10-19 21:46:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.7163 [2026-10-19 21:46:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.5035 [2026-10-19 21:47:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.4119 [2026-10-19 21:47:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.5712 [2026-10-19 21:47:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.3693 [2026-10-19 21:47:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.5232 [2026-10-19 21:47:28][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,535 | MFU: 1.8% [2026-10-19 21:47:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.1603 [2026-10-19 21:47:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.6494 [2026-10-19 21:47:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.1348 [2026-10-19 21:47:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.4625 [2026-10-19 21:47:33][INFO][training.pretrain_engine] [Ph2|Step 78] loss=9.6625 | ppl=15717.34 | aux=0.02561 | lr=7.81e-05 | gnorm=1.398 | tok/s=29,273 | MFU=7.0% | step=143282ms | tokens=0.089B | mem=4.2/32.7GB | cost=$40.95/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 19:17 (141.5h) | Phase3: SKIPPED | SFT: Oct 26 00:01 (146.2h) | DPO (Final): Oct 26 00:35 (146.8h) [2026-10-19 21:47:33][INFO][training.pretrain_engine] Starting global step 79 (accumulating 32 micro-batches)... [2026-10-19 21:47:33][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 21:47:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.5940 [2026-10-19 21:47:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.7687 [2026-10-19 21:47:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.7648 [2026-10-19 21:47:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.6400 [2026-10-19 21:48:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.6683 [2026-10-19 21:48:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.0652 [2026-10-19 21:48:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.4857 [2026-10-19 21:48:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.6581 [2026-10-19 21:48:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.3110 [2026-10-19 21:48:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.4798 [2026-10-19 21:48:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.4530 [2026-10-19 21:48:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.4863 [2026-10-19 21:48:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.2770 [2026-10-19 21:48:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.3491 [2026-10-19 21:48:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.1956 [2026-10-19 21:48:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.2543 [2026-10-19 21:49:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.5977 [2026-10-19 21:49:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.4220 [2026-10-19 21:49:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.4997 [2026-10-19 21:49:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.4580 [2026-10-19 21:49:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.4885 [2026-10-19 21:49:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.4435 [2026-10-19 21:49:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.2487 [2026-10-19 21:49:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=10.0579 [2026-10-19 21:49:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.1923 [2026-10-19 21:49:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.6878 [2026-10-19 21:49:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.2660 [2026-10-19 21:49:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.3296 [2026-10-19 21:49:52][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,531 | MFU: 1.8% [2026-10-19 21:49:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.5676 [2026-10-19 21:49:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.3637 [2026-10-19 21:49:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.4578 [2026-10-19 21:49:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.4831 [2026-10-19 21:49:57][INFO][training.pretrain_engine] [Ph2|Step 79] loss=9.6121 | ppl=14945.26 | aux=0.02565 | lr=7.87e-05 | gnorm=1.375 | tok/s=29,272 | MFU=7.0% | step=143288ms | tokens=0.090B | mem=4.2/32.7GB | cost=$41.48/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 19:19 (141.5h) | Phase3: SKIPPED | SFT: Oct 26 00:03 (146.2h) | DPO (Final): Oct 26 00:38 (146.8h) [2026-10-19 21:49:57][INFO][training.pretrain_engine] Starting global step 80 (accumulating 32 micro-batches)... [2026-10-19 21:49:57][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 21:50:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.3207 [2026-10-19 21:50:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.3495 [2026-10-19 21:50:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.2506 [2026-10-19 21:50:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.3020 [2026-10-19 21:50:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.2440 [2026-10-19 21:50:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.2197 [2026-10-19 21:50:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.4261 [2026-10-19 21:50:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.3393 [2026-10-19 21:50:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.8185 [2026-10-19 21:50:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.3579 [2026-10-19 21:50:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.3817 [2026-10-19 21:50:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.4811 [2026-10-19 21:51:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.3488 [2026-10-19 21:51:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.4704 [2026-10-19 21:51:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.4385 [2026-10-19 21:51:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.2284 [2026-10-19 21:51:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.4735 [2026-10-19 21:51:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.3785 [2026-10-19 21:51:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.4769 [2026-10-19 21:51:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.6836 [2026-10-19 21:51:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.1893 [2026-10-19 21:51:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.3215 [2026-10-19 21:51:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.0641 [2026-10-19 21:51:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.0362 [2026-10-19 21:52:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.9979 [2026-10-19 21:52:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.0562 [2026-10-19 21:52:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.0607 [2026-10-19 21:52:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.1171 [2026-10-19 21:52:16][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,536 | MFU: 1.8% [2026-10-19 21:52:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.4184 [2026-10-19 21:52:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.3188 [2026-10-19 21:52:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.3041 [2026-10-19 21:52:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.3520 [2026-10-19 21:52:20][INFO][training.pretrain_engine] [Ph2|Step 80] loss=9.5755 | ppl=14408.03 | aux=0.02555 | lr=7.93e-05 | gnorm=1.305 | tok/s=29,271 | MFU=7.0% | step=143293ms | tokens=0.091B | mem=4.2/32.7GB | cost=$42.00/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 19:21 (141.5h) | Phase3: SKIPPED | SFT: Oct 26 00:06 (146.2h) | DPO (Final): Oct 26 00:40 (146.8h) [2026-10-19 21:52:20][INFO][training.pretrain_engine] Starting global step 81 (accumulating 32 micro-batches)... [2026-10-19 21:52:20][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 21:52:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.8262 [2026-10-19 21:52:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.5739 [2026-10-19 21:52:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.5484 [2026-10-19 21:52:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.3935 [2026-10-19 21:52:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.8447 [2026-10-19 21:52:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.7412 [2026-10-19 21:52:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=10.0882 [2026-10-19 21:52:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.3554 [2026-10-19 21:53:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.4599 [2026-10-19 21:53:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.3904 [2026-10-19 21:53:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.4656 [2026-10-19 21:53:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.6500 [2026-10-19 21:53:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.4310 [2026-10-19 21:53:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.5460 [2026-10-19 21:53:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.4805 [2026-10-19 21:53:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.3848 [2026-10-19 21:53:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.5221 [2026-10-19 21:53:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.6707 [2026-10-19 21:53:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.8034 [2026-10-19 21:53:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.6445 [2026-10-19 21:54:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.7216 [2026-10-19 21:54:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.7984 [2026-10-19 21:54:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.5469 [2026-10-19 21:54:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.4999 [2026-10-19 21:54:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.6982 [2026-10-19 21:54:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.5189 [2026-10-19 21:54:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.4071 [2026-10-19 21:54:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.6604 [2026-10-19 21:54:40][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,525 | MFU: 1.8% [2026-10-19 21:54:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.3546 [2026-10-19 21:54:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.3193 [2026-10-19 21:54:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.4492 [2026-10-19 21:54:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.2963 [2026-10-19 21:54:44][INFO][training.pretrain_engine] [Ph2|Step 81] loss=9.5789 | ppl=14457.15 | aux=0.02545 | lr=7.98e-05 | gnorm=1.312 | tok/s=29,269 | MFU=7.0% | step=143300ms | tokens=0.092B | mem=4.2/32.7GB | cost=$42.53/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 19:23 (141.5h) | Phase3: SKIPPED | SFT: Oct 26 00:08 (146.2h) | DPO (Final): Oct 26 00:42 (146.8h) [2026-10-19 21:54:48][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step81 | Tokens: 0.092B | Spent: $42.54 | Remaining: $357.46 [2026-10-19 21:54:48][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step81 [2026-10-19 21:54:48][INFO][training.pretrain_engine] Starting global step 82 (accumulating 32 micro-batches)... [2026-10-19 21:54:48][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 21:54:48][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-19 21:54:48][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-19 21:54:48][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-19 21:54:48][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step81 to Phase-Technologies/forge-500m-pretrain (path: phase2_step81)... [2026-10-19 21:54:48][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-19 21:54:59][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-19 21:54:59][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step81 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-19 21:55:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.4995 [2026-10-19 21:55:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.2058 [2026-10-19 21:55:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.3006 [2026-10-19 21:55:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.2748 [2026-10-19 21:55:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.4300 [2026-10-19 21:55:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.3502 [2026-10-19 21:55:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.3627 [2026-10-19 21:55:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.2261 [2026-10-19 21:55:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.3517 [2026-10-19 21:55:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.3963 [2026-10-19 21:55:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.4396 [2026-10-19 21:55:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.3503 [2026-10-19 21:55:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.4412 [2026-10-19 21:55:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.4279 [2026-10-19 21:55:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.5661 [2026-10-19 21:55:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.4294 [2026-10-19 21:56:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.4790 [2026-10-19 21:56:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.4868 [2026-10-19 21:56:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.3716 [2026-10-19 21:56:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.2817 [2026-10-19 21:56:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.5091 [2026-10-19 21:56:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.7320 [2026-10-19 21:56:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.4352 [2026-10-19 21:56:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.4290 [2026-10-19 21:56:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.7269 [2026-10-19 21:56:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.9006 [2026-10-19 21:56:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.7290 [2026-10-19 21:56:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.8529 [2026-10-19 21:57:06][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,555 | MFU: 1.8% [2026-10-19 21:57:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.0230 [2026-10-19 21:57:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.9622 [2026-10-19 21:57:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.8634 [2026-10-19 21:57:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.8896 [2026-10-19 21:57:11][INFO][training.pretrain_engine] [Ph2|Step 82] loss=9.5433 | ppl=13951.40 | aux=0.02555 | lr=8.04e-05 | gnorm=1.234 | tok/s=29,269 | MFU=7.0% | step=143300ms | tokens=0.093B | mem=4.2/32.7GB | cost=$43.07/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 19:25 (141.5h) | Phase3: SKIPPED | SFT: Oct 26 00:10 (146.2h) | DPO (Final): Oct 26 00:44 (146.8h) [2026-10-19 21:57:11][INFO][training.pretrain_engine] Starting global step 83 (accumulating 32 micro-batches)... [2026-10-19 21:57:11][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 21:57:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.9160 [2026-10-19 21:57:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.7477 [2026-10-19 21:57:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.8982 [2026-10-19 21:57:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.0872 [2026-10-19 21:57:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.4182 [2026-10-19 21:57:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.3271 [2026-10-19 21:57:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.5783 [2026-10-19 21:57:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.9451 [2026-10-19 21:58:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.8018 [2026-10-19 21:58:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.4282 [2026-10-19 21:58:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.5208 [2026-10-19 21:58:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.6573 [2026-10-19 21:58:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.9045 [2026-10-19 21:58:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.4209 [2026-10-19 21:58:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.2821 [2026-10-19 21:58:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.5342 [2026-10-19 21:58:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.3607 [2026-10-19 21:58:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.6040 [2026-10-19 21:58:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.6919 [2026-10-19 21:58:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.4726 [2026-10-19 21:58:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.1992 [2026-10-19 21:58:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.2941 [2026-10-19 21:58:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.1998 [2026-10-19 21:58:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.2730 [2026-10-19 21:59:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.5516 [2026-10-19 21:59:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.6612 [2026-10-19 21:59:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.6496 [2026-10-19 21:59:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.5175 [2026-10-19 21:59:30][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,556 | MFU: 1.8% [2026-10-19 21:59:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.3847 [2026-10-19 21:59:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.5292 [2026-10-19 21:59:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.5269 [2026-10-19 21:59:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.4428 [2026-10-19 21:59:34][INFO][training.pretrain_engine] [Ph2|Step 83] loss=9.5467 | ppl=13997.97 | aux=0.02545 | lr=8.10e-05 | gnorm=1.281 | tok/s=29,269 | MFU=7.0% | step=143300ms | tokens=0.094B | mem=4.2/32.7GB | cost=$43.59/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 19:27 (141.5h) | Phase3: SKIPPED | SFT: Oct 26 00:11 (146.2h) | DPO (Final): Oct 26 00:46 (146.8h) [2026-10-19 21:59:34][INFO][training.pretrain_engine] Starting global step 84 (accumulating 32 micro-batches)... [2026-10-19 21:59:34][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 21:59:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.2342 [2026-10-19 21:59:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.4184 [2026-10-19 21:59:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.4987 [2026-10-19 21:59:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.3835 [2026-10-19 22:00:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.5003 [2026-10-19 22:00:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.4868 [2026-10-19 22:00:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.7441 [2026-10-19 22:00:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.2471 [2026-10-19 22:00:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.1650 [2026-10-19 22:00:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.0847 [2026-10-19 22:00:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.5799 [2026-10-19 22:00:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.1036 [2026-10-19 22:00:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.4627 [2026-10-19 22:00:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.2637 [2026-10-19 22:00:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.9806 [2026-10-19 22:00:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.4699 [2026-10-19 22:01:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.7028 [2026-10-19 22:01:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.4299 [2026-10-19 22:01:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.7063 [2026-10-19 22:01:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.0762 [2026-10-19 22:01:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.3205 [2026-10-19 22:01:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.3287 [2026-10-19 22:01:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.3991 [2026-10-19 22:01:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.4905 [2026-10-19 22:01:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.2855 [2026-10-19 22:01:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.3338 [2026-10-19 22:01:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.2633 [2026-10-19 22:01:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.2644 [2026-10-19 22:01:52][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,578 | MFU: 1.8% [2026-10-19 22:01:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.2293 [2026-10-19 22:01:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.1641 [2026-10-19 22:01:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.1069 [2026-10-19 22:01:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.2017 [2026-10-19 22:01:57][INFO][training.pretrain_engine] [Ph2|Step 84] loss=9.4295 | ppl=12450.64 | aux=0.02553 | lr=8.15e-05 | gnorm=1.297 | tok/s=29,270 | MFU=7.0% | step=143298ms | tokens=0.095B | mem=4.2/32.7GB | cost=$44.11/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 19:28 (141.4h) | Phase3: SKIPPED | SFT: Oct 26 00:13 (146.2h) | DPO (Final): Oct 26 00:47 (146.8h) [2026-10-19 22:01:57][INFO][training.pretrain_engine] Starting global step 85 (accumulating 32 micro-batches)... [2026-10-19 22:01:57][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 22:02:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.0758 [2026-10-19 22:02:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.2616 [2026-10-19 22:02:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.5838 [2026-10-19 22:02:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.3658 [2026-10-19 22:02:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.4294 [2026-10-19 22:02:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.6128 [2026-10-19 22:02:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.3350 [2026-10-19 22:02:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.0402 [2026-10-19 22:02:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.7475 [2026-10-19 22:02:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.8592 [2026-10-19 22:02:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=10.0523 [2026-10-19 22:02:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.8680 [2026-10-19 22:03:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.3114 [2026-10-19 22:03:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.8586 [2026-10-19 22:03:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.3332 [2026-10-19 22:03:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.7146 [2026-10-19 22:03:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.3602 [2026-10-19 22:03:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.4897 [2026-10-19 22:03:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.3067 [2026-10-19 22:03:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.2157 [2026-10-19 22:03:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.0771 [2026-10-19 22:03:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.0700 [2026-10-19 22:03:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.0443 [2026-10-19 22:03:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.2001 [2026-10-19 22:04:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.9227 [2026-10-19 22:04:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.7476 [2026-10-19 22:04:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.8466 [2026-10-19 22:04:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.3930 [2026-10-19 22:04:16][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,546 | MFU: 1.8% [2026-10-19 22:04:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.7415 [2026-10-19 22:04:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.3298 [2026-10-19 22:04:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.8533 [2026-10-19 22:04:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.7539 [2026-10-19 22:04:21][INFO][training.pretrain_engine] [Ph2|Step 85] loss=9.5381 | ppl=13879.14 | aux=0.02540 | lr=8.21e-05 | gnorm=1.195 | tok/s=29,269 | MFU=7.0% | step=143300ms | tokens=0.096B | mem=4.2/32.7GB | cost=$44.64/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 19:30 (141.4h) | Phase3: SKIPPED | SFT: Oct 26 00:15 (146.2h) | DPO (Final): Oct 26 00:49 (146.8h) [2026-10-19 22:04:21][INFO][training.pretrain_engine] Starting global step 86 (accumulating 32 micro-batches)... [2026-10-19 22:04:21][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 22:04:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.6138 [2026-10-19 22:04:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.9058 [2026-10-19 22:04:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.5368 [2026-10-19 22:04:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.4526 [2026-10-19 22:04:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.3283 [2026-10-19 22:04:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.2558 [2026-10-19 22:04:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.4935 [2026-10-19 22:04:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.4895 [2026-10-19 22:05:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.2159 [2026-10-19 22:05:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.4766 [2026-10-19 22:05:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.4945 [2026-10-19 22:05:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.6777 [2026-10-19 22:05:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.3153 [2026-10-19 22:05:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.3514 [2026-10-19 22:05:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.2586 [2026-10-19 22:05:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.1587 [2026-10-19 22:05:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.1844 [2026-10-19 22:05:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.4159 [2026-10-19 22:05:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.2981 [2026-10-19 22:05:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.1671 [2026-10-19 22:06:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.4723 [2026-10-19 22:06:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.4751 [2026-10-19 22:06:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.4437 [2026-10-19 22:06:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.2616 [2026-10-19 22:06:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.9002 [2026-10-19 22:06:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.5898 [2026-10-19 22:06:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.6759 [2026-10-19 22:06:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.6763 [2026-10-19 22:06:40][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,556 | MFU: 1.8% [2026-10-19 22:06:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.3870 [2026-10-19 22:06:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.3926 [2026-10-19 22:06:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.2890 [2026-10-19 22:06:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.2475 [2026-10-19 22:06:44][INFO][training.pretrain_engine] [Ph2|Step 86] loss=9.4999 | ppl=13357.87 | aux=0.02550 | lr=8.27e-05 | gnorm=1.133 | tok/s=29,269 | MFU=7.0% | step=143300ms | tokens=0.098B | mem=4.2/32.7GB | cost=$45.16/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 19:32 (141.4h) | Phase3: SKIPPED | SFT: Oct 26 00:17 (146.2h) | DPO (Final): Oct 26 00:51 (146.7h) [2026-10-19 22:06:44][INFO][training.pretrain_engine] Starting global step 87 (accumulating 32 micro-batches)... [2026-10-19 22:06:44][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 22:07:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.8795 [2026-10-19 22:07:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.8999 [2026-10-19 22:07:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.7111 [2026-10-19 22:07:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.7230 [2026-10-19 22:07:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.2650 [2026-10-19 22:07:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.3571 [2026-10-19 22:07:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.2724 [2026-10-19 22:07:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.2878 [2026-10-19 22:07:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.7453 [2026-10-19 22:07:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.7994 [2026-10-19 22:07:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.7402 [2026-10-19 22:07:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.6352 [2026-10-19 22:07:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.3788 [2026-10-19 22:07:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.7411 [2026-10-19 22:07:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=10.2469 [2026-10-19 22:07:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.7055 [2026-10-19 22:08:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.3816 [2026-10-19 22:08:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.5628 [2026-10-19 22:08:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.2172 [2026-10-19 22:08:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.2717 [2026-10-19 22:08:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.5497 [2026-10-19 22:08:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.4050 [2026-10-19 22:08:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.7082 [2026-10-19 22:08:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.2530 [2026-10-19 22:08:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.3412 [2026-10-19 22:08:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.3023 [2026-10-19 22:08:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.4450 [2026-10-19 22:08:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.7312 [2026-10-19 22:09:03][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,531 | MFU: 1.8% [2026-10-19 22:09:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.4461 [2026-10-19 22:09:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.1963 [2026-10-19 22:09:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.3658 [2026-10-19 22:09:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.0611 [2026-10-19 22:09:08][INFO][training.pretrain_engine] [Ph2|Step 87] loss=9.3779 | ppl=11824.52 | aux=0.02534 | lr=8.32e-05 | gnorm=1.266 | tok/s=29,268 | MFU=7.0% | step=143305ms | tokens=0.099B | mem=4.2/32.7GB | cost=$45.69/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 19:34 (141.4h) | Phase3: SKIPPED | SFT: Oct 26 00:19 (146.2h) | DPO (Final): Oct 26 00:53 (146.7h) [2026-10-19 22:09:08][INFO][training.pretrain_engine] Starting global step 88 (accumulating 32 micro-batches)... [2026-10-19 22:09:08][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 22:09:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.4384 [2026-10-19 22:09:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.6415 [2026-10-19 22:09:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.3136 [2026-10-19 22:09:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.2993 [2026-10-19 22:09:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.0767 [2026-10-19 22:09:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.2324 [2026-10-19 22:09:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.1707 [2026-10-19 22:09:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.5194 [2026-10-19 22:10:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.5207 [2026-10-19 22:10:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.5241 [2026-10-19 22:10:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.4446 [2026-10-19 22:10:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.5389 [2026-10-19 22:10:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.6847 [2026-10-19 22:10:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.3711 [2026-10-19 22:10:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.4849 [2026-10-19 22:10:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.2719 [2026-10-19 22:10:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.1197 [2026-10-19 22:10:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.1314 [2026-10-19 22:10:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.0952 [2026-10-19 22:10:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.2063 [2026-10-19 22:10:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.2906 [2026-10-19 22:10:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.1391 [2026-10-19 22:10:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.4872 [2026-10-19 22:10:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.3350 [2026-10-19 22:11:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.3088 [2026-10-19 22:11:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.7964 [2026-10-19 22:11:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=10.0522 [2026-10-19 22:11:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.6078 [2026-10-19 22:11:27][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,549 | MFU: 1.8% [2026-10-19 22:11:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.2569 [2026-10-19 22:11:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.2670 [2026-10-19 22:11:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.9951 [2026-10-19 22:11:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.1381 [2026-10-19 22:11:31][INFO][training.pretrain_engine] [Ph2|Step 88] loss=9.3273 | ppl=11240.83 | aux=0.02540 | lr=8.38e-05 | gnorm=1.445 | tok/s=29,268 | MFU=7.0% | step=143307ms | tokens=0.100B | mem=4.2/32.7GB | cost=$46.21/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 19:36 (141.4h) | Phase3: SKIPPED | SFT: Oct 26 00:21 (146.2h) | DPO (Final): Oct 26 00:55 (146.7h) [2026-10-19 22:11:31][INFO][training.pretrain_engine] Starting global step 89 (accumulating 32 micro-batches)... [2026-10-19 22:11:31][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 22:11:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=10.2294 [2026-10-19 22:11:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.2567 [2026-10-19 22:11:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.4290 [2026-10-19 22:11:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.2995 [2026-10-19 22:12:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.9951 [2026-10-19 22:12:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.9152 [2026-10-19 22:12:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.2260 [2026-10-19 22:12:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.0841 [2026-10-19 22:12:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.2388 [2026-10-19 22:12:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.3790 [2026-10-19 22:12:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.7823 [2026-10-19 22:12:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.8170 [2026-10-19 22:12:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.0928 [2026-10-19 22:12:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.2576 [2026-10-19 22:12:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.9011 [2026-10-19 22:12:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.0752 [2026-10-19 22:13:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.2018 [2026-10-19 22:13:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.1359 [2026-10-19 22:13:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.3490 [2026-10-19 22:13:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.1566 [2026-10-19 22:13:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.9457 [2026-10-19 22:13:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.0381 [2026-10-19 22:13:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.3846 [2026-10-19 22:13:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.2147 [2026-10-19 22:13:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.9701 [2026-10-19 22:13:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.8242 [2026-10-19 22:13:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.2972 [2026-10-19 22:13:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.5306 [2026-10-19 22:13:50][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,574 | MFU: 1.8% [2026-10-19 22:13:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.4753 [2026-10-19 22:13:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.5116 [2026-10-19 22:13:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=10.0128 [2026-10-19 22:13:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.4120 [2026-10-19 22:13:54][INFO][training.pretrain_engine] [Ph2|Step 89] loss=9.3570 | ppl=11580.01 | aux=0.02547 | lr=8.44e-05 | gnorm=1.500 | tok/s=29,269 | MFU=7.0% | step=143303ms | tokens=0.101B | mem=4.2/32.7GB | cost=$46.73/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 19:38 (141.4h) | Phase3: SKIPPED | SFT: Oct 26 00:22 (146.1h) | DPO (Final): Oct 26 00:57 (146.7h) [2026-10-19 22:13:54][INFO][training.pretrain_engine] Starting global step 90 (accumulating 32 micro-batches)... [2026-10-19 22:13:54][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 22:14:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.1748 [2026-10-19 22:14:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.2213 [2026-10-19 22:14:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.9869 [2026-10-19 22:14:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.8936 [2026-10-19 22:14:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.2697 [2026-10-19 22:14:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.0207 [2026-10-19 22:14:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.1089 [2026-10-19 22:14:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.3349 [2026-10-19 22:14:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.2442 [2026-10-19 22:14:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.3661 [2026-10-19 22:14:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.8610 [2026-10-19 22:14:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.4336 [2026-10-19 22:15:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.1249 [2026-10-19 22:15:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.2438 [2026-10-19 22:15:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.2128 [2026-10-19 22:15:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.8885 [2026-10-19 22:15:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.4665 [2026-10-19 22:15:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.5241 [2026-10-19 22:15:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.4209 [2026-10-19 22:15:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.5400 [2026-10-19 22:15:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.1913 [2026-10-19 22:15:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.0304 [2026-10-19 22:15:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.0680 [2026-10-19 22:15:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.3275 [2026-10-19 22:15:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.3346 [2026-10-19 22:15:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.1172 [2026-10-19 22:15:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.2298 [2026-10-19 22:15:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.9600 [2026-10-19 22:16:12][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,592 | MFU: 1.8% [2026-10-19 22:16:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.5760 [2026-10-19 22:16:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.1637 [2026-10-19 22:16:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.4181 [2026-10-19 22:16:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.4411 [2026-10-19 22:16:17][INFO][training.pretrain_engine] [Ph2|Step 90] loss=9.2880 | ppl=10807.68 | aux=0.02535 | lr=8.49e-05 | gnorm=1.211 | tok/s=29,269 | MFU=7.0% | step=143301ms | tokens=0.102B | mem=4.2/32.7GB | cost=$47.26/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 19:39 (141.4h) | Phase3: SKIPPED | SFT: Oct 26 00:24 (146.1h) | DPO (Final): Oct 26 00:58 (146.7h) [2026-10-19 22:16:20][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step90 | Tokens: 0.102B | Spent: $47.27 | Remaining: $352.73 [2026-10-19 22:16:20][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step90 [2026-10-19 22:16:20][INFO][training.pretrain_engine] Starting global step 91 (accumulating 32 micro-batches)... [2026-10-19 22:16:20][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 22:16:20][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-19 22:16:20][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-19 22:16:21][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-19 22:16:21][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step90 to Phase-Technologies/forge-500m-pretrain (path: phase2_step90)... [2026-10-19 22:16:21][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-19 22:16:31][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-19 22:16:31][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step90 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-19 22:16:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.6269 [2026-10-19 22:16:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.5287 [2026-10-19 22:16:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.6534 [2026-10-19 22:16:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.5651 [2026-10-19 22:16:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.9837 [2026-10-19 22:16:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.1603 [2026-10-19 22:16:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.0466 [2026-10-19 22:16:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.0556 [2026-10-19 22:17:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.5441 [2026-10-19 22:17:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.3304 [2026-10-19 22:17:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.6739 [2026-10-19 22:17:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.4140 [2026-10-19 22:17:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.3332 [2026-10-19 22:17:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.1838 [2026-10-19 22:17:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.1137 [2026-10-19 22:17:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.0977 [2026-10-19 22:17:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.4121 [2026-10-19 22:17:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.8883 [2026-10-19 22:17:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.0910 [2026-10-19 22:17:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.4635 [2026-10-19 22:18:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.2274 [2026-10-19 22:18:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.5374 [2026-10-19 22:18:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.2415 [2026-10-19 22:18:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.2272 [2026-10-19 22:18:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.7503 [2026-10-19 22:18:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.1034 [2026-10-19 22:18:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.6660 [2026-10-19 22:18:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.3132 [2026-10-19 22:18:38][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,592 | MFU: 1.8% [2026-10-19 22:18:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.7599 [2026-10-19 22:18:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.0412 [2026-10-19 22:18:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.0269 [2026-10-19 22:18:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.3236 [2026-10-19 22:18:43][INFO][training.pretrain_engine] [Ph2|Step 91] loss=9.2251 | ppl=10148.41 | aux=0.02532 | lr=8.55e-05 | gnorm=1.234 | tok/s=29,271 | MFU=7.0% | step=143293ms | tokens=0.103B | mem=4.2/32.7GB | cost=$47.79/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 19:41 (141.4h) | Phase3: SKIPPED | SFT: Oct 26 00:25 (146.1h) | DPO (Final): Oct 26 01:00 (146.7h) [2026-10-19 22:18:43][INFO][training.pretrain_engine] Starting global step 92 (accumulating 32 micro-batches)... [2026-10-19 22:18:43][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 22:19:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.9866 [2026-10-19 22:19:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.2844 [2026-10-19 22:19:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.0923 [2026-10-19 22:19:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.0536 [2026-10-19 22:19:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.1514 [2026-10-19 22:19:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.2601 [2026-10-19 22:19:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.9802 [2026-10-19 22:19:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.2830 [2026-10-19 22:19:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.8715 [2026-10-19 22:19:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.1071 [2026-10-19 22:19:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.2804 [2026-10-19 22:19:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.3922 [2026-10-19 22:19:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.2322 [2026-10-19 22:19:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.6789 [2026-10-19 22:19:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.2827 [2026-10-19 22:19:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.9933 [2026-10-19 22:20:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.5162 [2026-10-19 22:20:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.4649 [2026-10-19 22:20:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.4964 [2026-10-19 22:20:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.8625 [2026-10-19 22:20:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.9071 [2026-10-19 22:20:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.4603 [2026-10-19 22:20:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.9417 [2026-10-19 22:20:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.8118 [2026-10-19 22:20:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.3043 [2026-10-19 22:20:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.4083 [2026-10-19 22:20:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.5387 [2026-10-19 22:20:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.2913 [2026-10-19 22:21:02][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,554 | MFU: 1.8% [2026-10-19 22:21:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.9622 [2026-10-19 22:21:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.1568 [2026-10-19 22:21:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.8895 [2026-10-19 22:21:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.9502 [2026-10-19 22:21:06][INFO][training.pretrain_engine] [Ph2|Step 92] loss=9.2766 | ppl=10685.31 | aux=0.02537 | lr=8.61e-05 | gnorm=1.430 | tok/s=29,271 | MFU=7.0% | step=143294ms | tokens=0.104B | mem=4.2/32.7GB | cost=$48.31/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 19:42 (141.4h) | Phase3: SKIPPED | SFT: Oct 26 00:27 (146.1h) | DPO (Final): Oct 26 01:01 (146.7h) [2026-10-19 22:21:06][INFO][training.pretrain_engine] Starting global step 93 (accumulating 32 micro-batches)... [2026-10-19 22:21:06][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 22:21:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.0917 [2026-10-19 22:21:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.4167 [2026-10-19 22:21:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.4118 [2026-10-19 22:21:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.1066 [2026-10-19 22:21:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.0545 [2026-10-19 22:21:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.3910 [2026-10-19 22:21:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.9332 [2026-10-19 22:21:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.3665 [2026-10-19 22:22:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.0697 [2026-10-19 22:22:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.9372 [2026-10-19 22:22:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.9823 [2026-10-19 22:22:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.8350 [2026-10-19 22:22:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.1106 [2026-10-19 22:22:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.0504 [2026-10-19 22:22:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.1652 [2026-10-19 22:22:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.4277 [2026-10-19 22:22:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.3920 [2026-10-19 22:22:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.2896 [2026-10-19 22:22:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.6457 [2026-10-19 22:22:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.3710 [2026-10-19 22:22:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.6380 [2026-10-19 22:22:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.8422 [2026-10-19 22:22:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.0231 [2026-10-19 22:22:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.0617 [2026-10-19 22:23:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.8457 [2026-10-19 22:23:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.8365 [2026-10-19 22:23:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.9184 [2026-10-19 22:23:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.7979 [2026-10-19 22:23:25][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,544 | MFU: 1.8% [2026-10-19 22:23:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.1816 [2026-10-19 22:23:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.4892 [2026-10-19 22:23:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.5379 [2026-10-19 22:23:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.1668 [2026-10-19 22:23:30][INFO][training.pretrain_engine] [Ph2|Step 93] loss=9.3164 | ppl=11118.90 | aux=0.02542 | lr=8.66e-05 | gnorm=1.125 | tok/s=29,270 | MFU=7.0% | step=143296ms | tokens=0.105B | mem=4.2/32.7GB | cost=$48.84/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 19:44 (141.4h) | Phase3: SKIPPED | SFT: Oct 26 00:29 (146.1h) | DPO (Final): Oct 26 01:03 (146.7h) [2026-10-19 22:23:30][INFO][training.pretrain_engine] Starting global step 94 (accumulating 32 micro-batches)... [2026-10-19 22:23:30][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 22:23:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.0662 [2026-10-19 22:23:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.9507 [2026-10-19 22:23:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.0606 [2026-10-19 22:23:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.0840 [2026-10-19 22:24:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.1615 [2026-10-19 22:24:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.7000 [2026-10-19 22:24:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.2886 [2026-10-19 22:24:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.3327 [2026-10-19 22:24:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.9955 [2026-10-19 22:24:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.0418 [2026-10-19 22:24:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.3514 [2026-10-19 22:24:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.8420 [2026-10-19 22:24:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.0898 [2026-10-19 22:24:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.9446 [2026-10-19 22:24:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.9673 [2026-10-19 22:24:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.0328 [2026-10-19 22:24:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.3574 [2026-10-19 22:24:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.7016 [2026-10-19 22:24:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.5474 [2026-10-19 22:24:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.4021 [2026-10-19 22:25:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.9348 [2026-10-19 22:25:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.1224 [2026-10-19 22:25:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.9115 [2026-10-19 22:25:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.0702 [2026-10-19 22:25:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.9322 [2026-10-19 22:25:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.4116 [2026-10-19 22:25:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.0585 [2026-10-19 22:25:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.2575 [2026-10-19 22:25:48][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,571 | MFU: 1.8% [2026-10-19 22:25:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.4945 [2026-10-19 22:25:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.3904 [2026-10-19 22:25:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.4348 [2026-10-19 22:25:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.0244 [2026-10-19 22:25:53][INFO][training.pretrain_engine] [Ph2|Step 94] loss=9.2329 | ppl=10228.27 | aux=0.02531 | lr=8.72e-05 | gnorm=1.195 | tok/s=29,271 | MFU=7.0% | step=143293ms | tokens=0.106B | mem=4.2/32.7GB | cost=$49.36/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 19:46 (141.3h) | Phase3: SKIPPED | SFT: Oct 26 00:31 (146.1h) | DPO (Final): Oct 26 01:05 (146.7h) [2026-10-19 22:25:53][INFO][training.pretrain_engine] Starting global step 95 (accumulating 32 micro-batches)... [2026-10-19 22:25:53][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 22:26:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.2873 [2026-10-19 22:26:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.2810 [2026-10-19 22:26:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.2160 [2026-10-19 22:26:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.2933 [2026-10-19 22:26:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.1666 [2026-10-19 22:26:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.9052 [2026-10-19 22:26:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.1637 [2026-10-19 22:26:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.1028 [2026-10-19 22:26:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.3830 [2026-10-19 22:26:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.4578 [2026-10-19 22:26:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.4503 [2026-10-19 22:26:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.5607 [2026-10-19 22:27:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.1769 [2026-10-19 22:27:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.5366 [2026-10-19 22:27:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.9490 [2026-10-19 22:27:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.3649 [2026-10-19 22:27:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.1192 [2026-10-19 22:27:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.0137 [2026-10-19 22:27:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.1334 [2026-10-19 22:27:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.9018 [2026-10-19 22:27:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.7993 [2026-10-19 22:27:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.0692 [2026-10-19 22:27:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.1438 [2026-10-19 22:27:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.0854 [2026-10-19 22:27:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.0278 [2026-10-19 22:27:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.0129 [2026-10-19 22:27:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.9475 [2026-10-19 22:27:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.9279 [2026-10-19 22:28:11][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,611 | MFU: 1.8% [2026-10-19 22:28:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.5794 [2026-10-19 22:28:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.5403 [2026-10-19 22:28:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.5197 [2026-10-19 22:28:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.6610 [2026-10-19 22:28:16][INFO][training.pretrain_engine] [Ph2|Step 95] loss=9.1657 | ppl=9563.22 | aux=0.02535 | lr=8.78e-05 | gnorm=1.281 | tok/s=29,271 | MFU=7.0% | step=143292ms | tokens=0.107B | mem=4.2/32.7GB | cost=$49.88/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 19:48 (141.3h) | Phase3: SKIPPED | SFT: Oct 26 00:32 (146.1h) | DPO (Final): Oct 26 01:07 (146.6h) [2026-10-19 22:28:16][INFO][training.pretrain_engine] Starting global step 96 (accumulating 32 micro-batches)... [2026-10-19 22:28:16][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 22:28:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.2789 [2026-10-19 22:28:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.0763 [2026-10-19 22:28:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.1502 [2026-10-19 22:28:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.0571 [2026-10-19 22:28:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.5437 [2026-10-19 22:28:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.4505 [2026-10-19 22:28:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.6142 [2026-10-19 22:28:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.4137 [2026-10-19 22:29:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.9555 [2026-10-19 22:29:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.1330 [2026-10-19 22:29:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.1303 [2026-10-19 22:29:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.2216 [2026-10-19 22:29:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.6302 [2026-10-19 22:29:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.1796 [2026-10-19 22:29:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.4431 [2026-10-19 22:29:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.3333 [2026-10-19 22:29:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.8153 [2026-10-19 22:29:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.8161 [2026-10-19 22:29:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.8202 [2026-10-19 22:29:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.8276 [2026-10-19 22:30:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.4668 [2026-10-19 22:30:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.5210 [2026-10-19 22:30:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.6152 [2026-10-19 22:30:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.4132 [2026-10-19 22:30:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.1391 [2026-10-19 22:30:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.8029 [2026-10-19 22:30:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.1394 [2026-10-19 22:30:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.0316 [2026-10-19 22:30:34][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,582 | MFU: 1.8% [2026-10-19 22:30:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.9980 [2026-10-19 22:30:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.0129 [2026-10-19 22:30:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.8617 [2026-10-19 22:30:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.0050 [2026-10-19 22:30:39][INFO][training.pretrain_engine] [Ph2|Step 96] loss=9.1573 | ppl=9483.77 | aux=0.02535 | lr=8.83e-05 | gnorm=1.172 | tok/s=29,271 | MFU=7.0% | step=143292ms | tokens=0.108B | mem=4.2/32.7GB | cost=$50.41/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 19:50 (141.3h) | Phase3: SKIPPED | SFT: Oct 26 00:34 (146.1h) | DPO (Final): Oct 26 01:08 (146.6h) [2026-10-19 22:30:39][INFO][training.pretrain_engine] Starting global step 97 (accumulating 32 micro-batches)... [2026-10-19 22:30:39][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 22:30:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.1040 [2026-10-19 22:30:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.5113 [2026-10-19 22:30:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.6157 [2026-10-19 22:30:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.3698 [2026-10-19 22:31:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.0516 [2026-10-19 22:31:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.9252 [2026-10-19 22:31:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.0399 [2026-10-19 22:31:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.0852 [2026-10-19 22:31:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.7804 [2026-10-19 22:31:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.6802 [2026-10-19 22:31:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.5762 [2026-10-19 22:31:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.6085 [2026-10-19 22:31:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.3296 [2026-10-19 22:31:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.5650 [2026-10-19 22:31:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.5866 [2026-10-19 22:31:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.7375 [2026-10-19 22:32:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.2167 [2026-10-19 22:32:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.0845 [2026-10-19 22:32:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.0143 [2026-10-19 22:32:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.3450 [2026-10-19 22:32:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.3933 [2026-10-19 22:32:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.3884 [2026-10-19 22:32:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.4915 [2026-10-19 22:32:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.8226 [2026-10-19 22:32:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.3634 [2026-10-19 22:32:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.5703 [2026-10-19 22:32:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.1971 [2026-10-19 22:32:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.6319 [2026-10-19 22:32:59][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,529 | MFU: 1.8% [2026-10-19 22:33:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.9361 [2026-10-19 22:33:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.0079 [2026-10-19 22:33:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.8709 [2026-10-19 22:33:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.9034 [2026-10-19 22:33:03][INFO][training.pretrain_engine] [Ph2|Step 97] loss=9.1199 | ppl=9135.03 | aux=0.02529 | lr=8.89e-05 | gnorm=1.133 | tok/s=29,270 | MFU=7.0% | step=143297ms | tokens=0.109B | mem=4.2/32.7GB | cost=$50.93/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 19:52 (141.3h) | Phase3: SKIPPED | SFT: Oct 26 00:36 (146.1h) | DPO (Final): Oct 26 01:11 (146.6h) [2026-10-19 22:33:03][INFO][training.pretrain_engine] Starting global step 98 (accumulating 32 micro-batches)... [2026-10-19 22:33:03][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 22:33:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.6146 [2026-10-19 22:33:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.7938 [2026-10-19 22:33:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.9405 [2026-10-19 22:33:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.7207 [2026-10-19 22:33:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.9416 [2026-10-19 22:33:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.0701 [2026-10-19 22:33:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.8686 [2026-10-19 22:33:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.0899 [2026-10-19 22:33:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.9532 [2026-10-19 22:33:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.5096 [2026-10-19 22:33:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.4605 [2026-10-19 22:33:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.9097 [2026-10-19 22:34:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.8244 [2026-10-19 22:34:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.3411 [2026-10-19 22:34:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.1865 [2026-10-19 22:34:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.0921 [2026-10-19 22:34:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.9884 [2026-10-19 22:34:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.8782 [2026-10-19 22:34:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.9293 [2026-10-19 22:34:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.7786 [2026-10-19 22:34:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.4388 [2026-10-19 22:34:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.7557 [2026-10-19 22:34:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.8826 [2026-10-19 22:34:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.9516 [2026-10-19 22:35:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.0323 [2026-10-19 22:35:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.7738 [2026-10-19 22:35:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.8737 [2026-10-19 22:35:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.8035 [2026-10-19 22:35:22][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,540 | MFU: 1.8% [2026-10-19 22:35:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.9270 [2026-10-19 22:35:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.9599 [2026-10-19 22:35:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.9996 [2026-10-19 22:35:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.2219 [2026-10-19 22:35:27][INFO][training.pretrain_engine] [Ph2|Step 98] loss=9.1242 | ppl=9174.63 | aux=0.02541 | lr=8.95e-05 | gnorm=1.117 | tok/s=29,269 | MFU=7.0% | step=143300ms | tokens=0.110B | mem=4.2/32.7GB | cost=$51.46/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 19:54 (141.3h) | Phase3: SKIPPED | SFT: Oct 26 00:38 (146.1h) | DPO (Final): Oct 26 01:12 (146.6h) [2026-10-19 22:35:27][INFO][training.pretrain_engine] Starting global step 99 (accumulating 32 micro-batches)... [2026-10-19 22:35:27][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 22:35:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.8928 [2026-10-19 22:35:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.5353 [2026-10-19 22:35:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.7250 [2026-10-19 22:35:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.9124 [2026-10-19 22:36:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.2724 [2026-10-19 22:36:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.2994 [2026-10-19 22:36:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.8097 [2026-10-19 22:36:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.2040 [2026-10-19 22:36:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.0332 [2026-10-19 22:36:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.1118 [2026-10-19 22:36:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.5242 [2026-10-19 22:36:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.9789 [2026-10-19 22:36:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.0368 [2026-10-19 22:36:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.0584 [2026-10-19 22:36:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.8833 [2026-10-19 22:36:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.8787 [2026-10-19 22:36:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.7000 [2026-10-19 22:36:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.7085 [2026-10-19 22:36:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.9831 [2026-10-19 22:36:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.9527 [2026-10-19 22:37:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.4580 [2026-10-19 22:37:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.0919 [2026-10-19 22:37:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.5115 [2026-10-19 22:37:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.2675 [2026-10-19 22:37:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.6772 [2026-10-19 22:37:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.9851 [2026-10-19 22:37:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.1097 [2026-10-19 22:37:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.1536 [2026-10-19 22:37:46][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,551 | MFU: 1.8% [2026-10-19 22:37:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.2513 [2026-10-19 22:37:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.2973 [2026-10-19 22:37:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.1203 [2026-10-19 22:37:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.9253 [2026-10-19 22:37:50][INFO][training.pretrain_engine] [Ph2|Step 99] loss=9.0271 | ppl=8325.35 | aux=0.02524 | lr=9.00e-05 | gnorm=1.133 | tok/s=29,269 | MFU=7.0% | step=143301ms | tokens=0.111B | mem=4.2/32.7GB | cost=$51.98/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 19:55 (141.3h) | Phase3: SKIPPED | SFT: Oct 26 00:40 (146.0h) | DPO (Final): Oct 26 01:14 (146.6h) [2026-10-19 22:37:54][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step99 | Tokens: 0.111B | Spent: $52.00 | Remaining: $348.00 [2026-10-19 22:37:54][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step99 [2026-10-19 22:37:54][INFO][training.pretrain_engine] Starting global step 100 (accumulating 32 micro-batches)... [2026-10-19 22:37:54][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 22:37:54][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-19 22:37:54][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-19 22:37:54][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-19 22:37:54][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step99 to Phase-Technologies/forge-500m-pretrain (path: phase2_step99)... [2026-10-19 22:37:54][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-19 22:38:04][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-19 22:38:04][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step99 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-19 22:38:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.9936 [2026-10-19 22:38:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.0677 [2026-10-19 22:38:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.1811 [2026-10-19 22:38:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.3848 [2026-10-19 22:38:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.5170 [2026-10-19 22:38:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.6595 [2026-10-19 22:38:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.9926 [2026-10-19 22:38:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.7716 [2026-10-19 22:38:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.7616 [2026-10-19 22:38:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.8218 [2026-10-19 22:38:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.6353 [2026-10-19 22:38:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.8244 [2026-10-19 22:39:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.9583 [2026-10-19 22:39:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.9653 [2026-10-19 22:39:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.0751 [2026-10-19 22:39:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.9092 [2026-10-19 22:39:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.5581 [2026-10-19 22:39:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.6898 [2026-10-19 22:39:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.6250 [2026-10-19 22:39:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.6127 [2026-10-19 22:39:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.8056 [2026-10-19 22:39:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.2283 [2026-10-19 22:39:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.8747 [2026-10-19 22:39:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.7496 [2026-10-19 22:39:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.8718 [2026-10-19 22:39:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.0854 [2026-10-19 22:39:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.4632 [2026-10-19 22:39:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.1488 [2026-10-19 22:40:12][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,557 | MFU: 1.8% [2026-10-19 22:40:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.7822 [2026-10-19 22:40:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.9819 [2026-10-19 22:40:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.3065 [2026-10-19 22:40:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.2292 [2026-10-19 22:40:17][INFO][training.pretrain_engine] [Ph2|Step 100] loss=9.0521 | ppl=8536.08 | aux=0.02533 | lr=9.06e-05 | gnorm=1.102 | tok/s=29,269 | MFU=7.0% | step=143300ms | tokens=0.112B | mem=4.2/32.7GB | cost=$52.52/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 19:57 (141.3h) | Phase3: SKIPPED | SFT: Oct 26 00:42 (146.0h) | DPO (Final): Oct 26 01:16 (146.6h) [2026-10-19 22:40:17][INFO][training.pretrain_engine] Starting global step 101 (accumulating 32 micro-batches)... [2026-10-19 22:40:17][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 22:40:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.5743 [2026-10-19 22:40:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.1547 [2026-10-19 22:40:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.1888 [2026-10-19 22:40:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.2635 [2026-10-19 22:40:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.3566 [2026-10-19 22:40:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.2894 [2026-10-19 22:40:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.1256 [2026-10-19 22:40:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.2907 [2026-10-19 22:41:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.8777 [2026-10-19 22:41:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.2639 [2026-10-19 22:41:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.6793 [2026-10-19 22:41:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.7959 [2026-10-19 22:41:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.6971 [2026-10-19 22:41:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.5818 [2026-10-19 22:41:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.9371 [2026-10-19 22:41:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.7445 [2026-10-19 22:41:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.7184 [2026-10-19 22:41:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.0103 [2026-10-19 22:41:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.0189 [2026-10-19 22:41:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.6475 [2026-10-19 22:42:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.6999 [2026-10-19 22:42:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.8185 [2026-10-19 22:42:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.7939 [2026-10-19 22:42:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.9039 [2026-10-19 22:42:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.8791 [2026-10-19 22:42:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.9339 [2026-10-19 22:42:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.0516 [2026-10-19 22:42:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.8074 [2026-10-19 22:42:35][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,597 | MFU: 1.8% [2026-10-19 22:42:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.0189 [2026-10-19 22:42:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.7775 [2026-10-19 22:42:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.7372 [2026-10-19 22:42:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.9746 [2026-10-19 22:42:40][INFO][training.pretrain_engine] [Ph2|Step 101] loss=9.0144 | ppl=8220.25 | aux=0.02537 | lr=9.12e-05 | gnorm=1.102 | tok/s=29,282 | MFU=7.0% | step=143238ms | tokens=0.113B | mem=4.2/32.7GB | cost=$53.04/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 19:55 (141.2h) | Phase3: SKIPPED | SFT: Oct 26 00:40 (146.0h) | DPO (Final): Oct 26 01:14 (146.5h) [2026-10-19 22:42:40][INFO][training.pretrain_engine] Starting global step 102 (accumulating 32 micro-batches)... [2026-10-19 22:42:40][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 22:42:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.9295 [2026-10-19 22:42:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.9657 [2026-10-19 22:42:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.3303 [2026-10-19 22:42:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.9378 [2026-10-19 22:43:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.9072 [2026-10-19 22:43:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.4345 [2026-10-19 22:43:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.9929 [2026-10-19 22:43:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.8687 [2026-10-19 22:43:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.7340 [2026-10-19 22:43:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.8461 [2026-10-19 22:43:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.7828 [2026-10-19 22:43:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.9411 [2026-10-19 22:43:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.1905 [2026-10-19 22:43:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.4556 [2026-10-19 22:43:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.4788 [2026-10-19 22:43:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.4997 [2026-10-19 22:44:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.9425 [2026-10-19 22:44:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.6702 [2026-10-19 22:44:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.7103 [2026-10-19 22:44:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.6257 [2026-10-19 22:44:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.2951 [2026-10-19 22:44:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.1489 [2026-10-19 22:44:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.2900 [2026-10-19 22:44:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.1807 [2026-10-19 22:44:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.2762 [2026-10-19 22:44:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.5614 [2026-10-19 22:44:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.3376 [2026-10-19 22:44:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.3357 [2026-10-19 22:44:59][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,578 | MFU: 1.8% [2026-10-19 22:45:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.2752 [2026-10-19 22:45:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.8213 [2026-10-19 22:45:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.6493 [2026-10-19 22:45:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.7209 [2026-10-19 22:45:04][INFO][training.pretrain_engine] [Ph2|Step 102] loss=9.0607 | ppl=8610.01 | aux=0.02530 | lr=9.17e-05 | gnorm=1.164 | tok/s=29,281 | MFU=7.0% | step=143242ms | tokens=0.114B | mem=4.2/32.7GB | cost=$53.57/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 19:57 (141.2h) | Phase3: SKIPPED | SFT: Oct 26 00:42 (146.0h) | DPO (Final): Oct 26 01:16 (146.5h) [2026-10-19 22:45:04][INFO][training.pretrain_engine] Starting global step 103 (accumulating 32 micro-batches)... [2026-10-19 22:45:04][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 22:45:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.8282 [2026-10-19 22:45:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.2993 [2026-10-19 22:45:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.6334 [2026-10-19 22:45:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.7071 [2026-10-19 22:45:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.6049 [2026-10-19 22:45:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.8850 [2026-10-19 22:45:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.0885 [2026-10-19 22:45:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.7087 [2026-10-19 22:45:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.1972 [2026-10-19 22:45:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.9288 [2026-10-19 22:45:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.0114 [2026-10-19 22:45:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.1104 [2026-10-19 22:46:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.7696 [2026-10-19 22:46:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.6867 [2026-10-19 22:46:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.9769 [2026-10-19 22:46:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.8043 [2026-10-19 22:46:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.8987 [2026-10-19 22:46:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.4601 [2026-10-19 22:46:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.0730 [2026-10-19 22:46:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.9665 [2026-10-19 22:46:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.6815 [2026-10-19 22:46:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.8859 [2026-10-19 22:46:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.7387 [2026-10-19 22:46:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.8632 [2026-10-19 22:47:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.1252 [2026-10-19 22:47:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.0779 [2026-10-19 22:47:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.3468 [2026-10-19 22:47:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.2351 [2026-10-19 22:47:22][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,556 | MFU: 1.8% [2026-10-19 22:47:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.8615 [2026-10-19 22:47:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.8949 [2026-10-19 22:47:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.8231 [2026-10-19 22:47:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.4423 [2026-10-19 22:47:27][INFO][training.pretrain_engine] [Ph2|Step 103] loss=8.9679 | ppl=7847.00 | aux=0.02532 | lr=9.23e-05 | gnorm=1.172 | tok/s=29,281 | MFU=7.0% | step=143245ms | tokens=0.115B | mem=4.2/32.7GB | cost=$54.09/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 19:59 (141.2h) | Phase3: SKIPPED | SFT: Oct 26 00:44 (145.9h) | DPO (Final): Oct 26 01:18 (146.5h) [2026-10-19 22:47:27][INFO][training.pretrain_engine] Starting global step 104 (accumulating 32 micro-batches)... [2026-10-19 22:47:27][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 22:47:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.2887 [2026-10-19 22:47:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.1380 [2026-10-19 22:47:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.5184 [2026-10-19 22:47:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.0319 [2026-10-19 22:48:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.8169 [2026-10-19 22:48:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.7708 [2026-10-19 22:48:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.7422 [2026-10-19 22:48:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.8863 [2026-10-19 22:48:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.8220 [2026-10-19 22:48:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.1466 [2026-10-19 22:48:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.8000 [2026-10-19 22:48:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.8997 [2026-10-19 22:48:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.3722 [2026-10-19 22:48:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.6127 [2026-10-19 22:48:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.2944 [2026-10-19 22:48:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.4982 [2026-10-19 22:48:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.5127 [2026-10-19 22:48:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.4343 [2026-10-19 22:48:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.3180 [2026-10-19 22:48:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.9063 [2026-10-19 22:49:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.9358 [2026-10-19 22:49:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.1711 [2026-10-19 22:49:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.0250 [2026-10-19 22:49:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.9039 [2026-10-19 22:49:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.2068 [2026-10-19 22:49:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.3772 [2026-10-19 22:49:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.8827 [2026-10-19 22:49:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.8836 [2026-10-19 22:49:46][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,553 | MFU: 1.8% [2026-10-19 22:49:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.4861 [2026-10-19 22:49:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.5117 [2026-10-19 22:49:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.6431 [2026-10-19 22:49:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.7645 [2026-10-19 22:49:50][INFO][training.pretrain_engine] [Ph2|Step 104] loss=8.9596 | ppl=7782.56 | aux=0.02532 | lr=9.29e-05 | gnorm=1.055 | tok/s=29,280 | MFU=7.0% | step=143247ms | tokens=0.116B | mem=4.2/32.7GB | cost=$54.62/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 20:01 (141.2h) | Phase3: SKIPPED | SFT: Oct 26 00:46 (145.9h) | DPO (Final): Oct 26 01:20 (146.5h) [2026-10-19 22:49:50][INFO][training.pretrain_engine] Starting global step 105 (accumulating 32 micro-batches)... [2026-10-19 22:49:50][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 22:50:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.9516 [2026-10-19 22:50:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.6156 [2026-10-19 22:50:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.9547 [2026-10-19 22:50:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.1898 [2026-10-19 22:50:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.6042 [2026-10-19 22:50:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.3979 [2026-10-19 22:50:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.5757 [2026-10-19 22:50:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.1195 [2026-10-19 22:50:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.1566 [2026-10-19 22:50:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.5886 [2026-10-19 22:50:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.2650 [2026-10-19 22:50:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.6401 [2026-10-19 22:51:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.2073 [2026-10-19 22:51:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.6270 [2026-10-19 22:51:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.8351 [2026-10-19 22:51:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.2487 [2026-10-19 22:51:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.6108 [2026-10-19 22:51:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.5008 [2026-10-19 22:51:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.7338 [2026-10-19 22:51:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.5454 [2026-10-19 22:51:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.9708 [2026-10-19 22:51:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.4073 [2026-10-19 22:51:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.1808 [2026-10-19 22:51:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.2368 [2026-10-19 22:51:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.3723 [2026-10-19 22:51:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.1433 [2026-10-19 22:51:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.6732 [2026-10-19 22:51:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.5562 [2026-10-19 22:52:09][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,570 | MFU: 1.8% [2026-10-19 22:52:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.0308 [2026-10-19 22:52:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.9842 [2026-10-19 22:52:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.9566 [2026-10-19 22:52:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.0368 [2026-10-19 22:52:14][INFO][training.pretrain_engine] [Ph2|Step 105] loss=8.9810 | ppl=7950.58 | aux=0.02526 | lr=9.34e-05 | gnorm=1.023 | tok/s=29,279 | MFU=7.0% | step=143253ms | tokens=0.117B | mem=4.2/32.7GB | cost=$55.14/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 20:03 (141.2h) | Phase3: SKIPPED | SFT: Oct 26 00:48 (145.9h) | DPO (Final): Oct 26 01:22 (146.5h) [2026-10-19 22:52:14][INFO][training.pretrain_engine] Starting global step 106 (accumulating 32 micro-batches)... [2026-10-19 22:52:14][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 22:52:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.3131 [2026-10-19 22:52:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.4702 [2026-10-19 22:52:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.3129 [2026-10-19 22:52:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.2963 [2026-10-19 22:52:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.5630 [2026-10-19 22:52:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.6402 [2026-10-19 22:52:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.7707 [2026-10-19 22:52:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.6087 [2026-10-19 22:53:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.9408 [2026-10-19 22:53:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.7924 [2026-10-19 22:53:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.8993 [2026-10-19 22:53:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.2081 [2026-10-19 22:53:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.5956 [2026-10-19 22:53:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.5879 [2026-10-19 22:53:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.6941 [2026-10-19 22:53:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.6522 [2026-10-19 22:53:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.5841 [2026-10-19 22:53:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.5640 [2026-10-19 22:53:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.7843 [2026-10-19 22:53:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.9255 [2026-10-19 22:54:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.6870 [2026-10-19 22:54:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.7068 [2026-10-19 22:54:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.4151 [2026-10-19 22:54:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.7258 [2026-10-19 22:54:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.6925 [2026-10-19 22:54:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.9939 [2026-10-19 22:54:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.7214 [2026-10-19 22:54:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.1759 [2026-10-19 22:54:32][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,571 | MFU: 1.8% [2026-10-19 22:54:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.7364 [2026-10-19 22:54:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.6406 [2026-10-19 22:54:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.6116 [2026-10-19 22:54:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.6613 [2026-10-19 22:54:37][INFO][training.pretrain_engine] [Ph2|Step 106] loss=8.8538 | ppl=7001.16 | aux=0.02527 | lr=9.40e-05 | gnorm=1.133 | tok/s=29,278 | MFU=7.0% | step=143258ms | tokens=0.118B | mem=4.2/32.7GB | cost=$55.66/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 20:05 (141.2h) | Phase3: SKIPPED | SFT: Oct 26 00:50 (145.9h) | DPO (Final): Oct 26 01:24 (146.5h) [2026-10-19 22:54:37][INFO][training.pretrain_engine] Starting global step 107 (accumulating 32 micro-batches)... [2026-10-19 22:54:37][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 22:54:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.1071 [2026-10-19 22:54:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.1984 [2026-10-19 22:54:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.2260 [2026-10-19 22:54:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.7857 [2026-10-19 22:55:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.6531 [2026-10-19 22:55:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.8601 [2026-10-19 22:55:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.7402 [2026-10-19 22:55:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.9817 [2026-10-19 22:55:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.6939 [2026-10-19 22:55:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.6931 [2026-10-19 22:55:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.5468 [2026-10-19 22:55:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.5079 [2026-10-19 22:55:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.4270 [2026-10-19 22:55:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.5448 [2026-10-19 22:55:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.7365 [2026-10-19 22:55:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.9615 [2026-10-19 22:56:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.8181 [2026-10-19 22:56:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.7846 [2026-10-19 22:56:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.6656 [2026-10-19 22:56:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.1428 [2026-10-19 22:56:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.0274 [2026-10-19 22:56:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.0529 [2026-10-19 22:56:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.8576 [2026-10-19 22:56:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.1499 [2026-10-19 22:56:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.0063 [2026-10-19 22:56:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.8905 [2026-10-19 22:56:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.7386 [2026-10-19 22:56:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.7036 [2026-10-19 22:56:55][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,589 | MFU: 1.8% [2026-10-19 22:57:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.3008 [2026-10-19 22:57:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.3022 [2026-10-19 22:57:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.5950 [2026-10-19 22:57:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.1530 [2026-10-19 22:57:01][INFO][training.pretrain_engine] [Ph2|Step 107] loss=8.8253 | ppl=6804.12 | aux=0.02534 | lr=9.46e-05 | gnorm=1.062 | tok/s=29,277 | MFU=7.0% | step=143263ms | tokens=0.120B | mem=4.2/32.7GB | cost=$56.19/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 20:08 (141.2h) | Phase3: SKIPPED | SFT: Oct 26 00:52 (145.9h) | DPO (Final): Oct 26 01:26 (146.5h) [2026-10-19 22:57:01][INFO][training.pretrain_engine] Starting global step 108 (accumulating 32 micro-batches)... [2026-10-19 22:57:01][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 22:57:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.3120 [2026-10-19 22:57:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.8421 [2026-10-19 22:57:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.0486 [2026-10-19 22:57:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.4702 [2026-10-19 22:57:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.4943 [2026-10-19 22:57:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.4897 [2026-10-19 22:57:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.5040 [2026-10-19 22:57:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.5760 [2026-10-19 22:57:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.0645 [2026-10-19 22:57:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.1767 [2026-10-19 22:57:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.1520 [2026-10-19 22:57:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.1111 [2026-10-19 22:58:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.8591 [2026-10-19 22:58:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.7729 [2026-10-19 22:58:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.3592 [2026-10-19 22:58:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.8653 [2026-10-19 22:58:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.8526 [2026-10-19 22:58:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.8860 [2026-10-19 22:58:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.0393 [2026-10-19 22:58:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.8017 [2026-10-19 22:58:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.6616 [2026-10-19 22:58:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.6114 [2026-10-19 22:58:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.7193 [2026-10-19 22:58:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.7203 [2026-10-19 22:59:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.2106 [2026-10-19 22:59:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.1647 [2026-10-19 22:59:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.1524 [2026-10-19 22:59:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.4161 [2026-10-19 22:59:19][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,584 | MFU: 1.8% [2026-10-19 22:59:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.1322 [2026-10-19 22:59:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.4287 [2026-10-19 22:59:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.7441 [2026-10-19 22:59:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.2589 [2026-10-19 22:59:23][INFO][training.pretrain_engine] [Ph2|Step 108] loss=8.8569 | ppl=7022.62 | aux=0.02524 | lr=9.51e-05 | gnorm=1.133 | tok/s=29,277 | MFU=7.0% | step=143262ms | tokens=0.121B | mem=4.2/32.7GB | cost=$56.71/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 20:09 (141.2h) | Phase3: SKIPPED | SFT: Oct 26 00:54 (145.9h) | DPO (Final): Oct 26 01:28 (146.5h) [2026-10-19 22:59:27][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step108 | Tokens: 0.121B | Spent: $56.72 | Remaining: $343.28 [2026-10-19 22:59:27][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step108 [2026-10-19 22:59:27][INFO][training.pretrain_engine] Starting global step 109 (accumulating 32 micro-batches)... [2026-10-19 22:59:27][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 22:59:27][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-19 22:59:27][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-19 22:59:27][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-19 22:59:27][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step108 to Phase-Technologies/forge-500m-pretrain (path: phase2_step108)... [2026-10-19 22:59:27][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-19 22:59:37][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-19 22:59:37][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step108 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-19 22:59:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.5131 [2026-10-19 22:59:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.6772 [2026-10-19 22:59:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.8961 [2026-10-19 22:59:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.0160 [2026-10-19 22:59:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.0716 [2026-10-19 22:59:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.7784 [2026-10-19 22:59:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.4772 [2026-10-19 23:00:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.2484 [2026-10-19 23:00:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.2993 [2026-10-19 23:00:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.1842 [2026-10-19 23:00:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.1957 [2026-10-19 23:00:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.1163 [2026-10-19 23:00:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.0910 [2026-10-19 23:00:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.0682 [2026-10-19 23:00:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.9403 [2026-10-19 23:00:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.0472 [2026-10-19 23:00:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.8992 [2026-10-19 23:00:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.7846 [2026-10-19 23:00:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.9035 [2026-10-19 23:00:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.0566 [2026-10-19 23:01:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.4957 [2026-10-19 23:01:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.5416 [2026-10-19 23:01:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.6924 [2026-10-19 23:01:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.6058 [2026-10-19 23:01:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.7646 [2026-10-19 23:01:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.6609 [2026-10-19 23:01:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.7622 [2026-10-19 23:01:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.7430 [2026-10-19 23:01:46][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,543 | MFU: 1.8% [2026-10-19 23:01:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.9033 [2026-10-19 23:01:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.0527 [2026-10-19 23:01:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.6161 [2026-10-19 23:01:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.6788 [2026-10-19 23:01:50][INFO][training.pretrain_engine] [Ph2|Step 109] loss=8.7947 | ppl=6599.35 | aux=0.02531 | lr=9.57e-05 | gnorm=1.289 | tok/s=29,276 | MFU=7.0% | step=143268ms | tokens=0.122B | mem=4.2/32.7GB | cost=$57.25/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 20:12 (141.2h) | Phase3: SKIPPED | SFT: Oct 26 00:56 (145.9h) | DPO (Final): Oct 26 01:30 (146.5h) [2026-10-19 23:01:50][INFO][training.pretrain_engine] Starting global step 110 (accumulating 32 micro-batches)... [2026-10-19 23:01:50][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 23:02:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.8161 [2026-10-19 23:02:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.8084 [2026-10-19 23:02:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.8772 [2026-10-19 23:02:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.5245 [2026-10-19 23:02:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.3729 [2026-10-19 23:02:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.1666 [2026-10-19 23:02:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.1475 [2026-10-19 23:02:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.5276 [2026-10-19 23:02:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.6848 [2026-10-19 23:02:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.8825 [2026-10-19 23:02:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.1547 [2026-10-19 23:02:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.4927 [2026-10-19 23:03:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.6036 [2026-10-19 23:03:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.4561 [2026-10-19 23:03:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.5103 [2026-10-19 23:03:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.4526 [2026-10-19 23:03:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.0331 [2026-10-19 23:03:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.5488 [2026-10-19 23:03:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.6936 [2026-10-19 23:03:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.4617 [2026-10-19 23:03:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.6880 [2026-10-19 23:03:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.6731 [2026-10-19 23:03:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.6955 [2026-10-19 23:03:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.7903 [2026-10-19 23:03:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.2465 [2026-10-19 23:03:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.6922 [2026-10-19 23:03:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.0392 [2026-10-19 23:03:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.7145 [2026-10-19 23:04:09][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,560 | MFU: 1.8% [2026-10-19 23:04:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.6015 [2026-10-19 23:04:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.4827 [2026-10-19 23:04:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.5118 [2026-10-19 23:04:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.7449 [2026-10-19 23:04:14][INFO][training.pretrain_engine] [Ph2|Step 110] loss=8.8242 | ppl=6796.94 | aux=0.02535 | lr=9.62e-05 | gnorm=1.414 | tok/s=29,275 | MFU=7.0% | step=143273ms | tokens=0.123B | mem=4.2/32.7GB | cost=$57.77/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 20:14 (141.2h) | Phase3: SKIPPED | SFT: Oct 26 00:58 (145.9h) | DPO (Final): Oct 26 01:32 (146.5h) [2026-10-19 23:04:14][INFO][training.pretrain_engine] Starting global step 111 (accumulating 32 micro-batches)... [2026-10-19 23:04:14][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 23:04:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.6167 [2026-10-19 23:04:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.5058 [2026-10-19 23:04:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.7180 [2026-10-19 23:04:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.5907 [2026-10-19 23:04:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.7550 [2026-10-19 23:04:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.5991 [2026-10-19 23:04:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.5369 [2026-10-19 23:04:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.6620 [2026-10-19 23:05:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.5805 [2026-10-19 23:05:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.6894 [2026-10-19 23:05:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.6598 [2026-10-19 23:05:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.1722 [2026-10-19 23:05:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.4616 [2026-10-19 23:05:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.6907 [2026-10-19 23:05:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.0960 [2026-10-19 23:05:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.6081 [2026-10-19 23:05:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.4539 [2026-10-19 23:05:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.4090 [2026-10-19 23:05:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.2343 [2026-10-19 23:05:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.5523 [2026-10-19 23:06:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.5236 [2026-10-19 23:06:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.0765 [2026-10-19 23:06:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.4111 [2026-10-19 23:06:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.0632 [2026-10-19 23:06:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.6834 [2026-10-19 23:06:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.7102 [2026-10-19 23:06:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.5430 [2026-10-19 23:06:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.5710 [2026-10-19 23:06:33][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,551 | MFU: 1.8% [2026-10-19 23:06:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.7738 [2026-10-19 23:06:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.7885 [2026-10-19 23:06:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.4868 [2026-10-19 23:06:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.7672 [2026-10-19 23:06:37][INFO][training.pretrain_engine] [Ph2|Step 111] loss=8.8398 | ppl=6903.29 | aux=0.02528 | lr=9.68e-05 | gnorm=0.996 | tok/s=29,275 | MFU=7.0% | step=143274ms | tokens=0.124B | mem=4.2/32.7GB | cost=$58.30/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 20:16 (141.2h) | Phase3: SKIPPED | SFT: Oct 26 01:00 (145.9h) | DPO (Final): Oct 26 01:34 (146.5h) [2026-10-19 23:06:37][INFO][training.pretrain_engine] Starting global step 112 (accumulating 32 micro-batches)... [2026-10-19 23:06:37][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 23:06:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.7923 [2026-10-19 23:06:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.9078 [2026-10-19 23:06:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.7913 [2026-10-19 23:06:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.2813 [2026-10-19 23:07:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.7911 [2026-10-19 23:07:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.6160 [2026-10-19 23:07:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.9006 [2026-10-19 23:07:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.1095 [2026-10-19 23:07:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.5600 [2026-10-19 23:07:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.9862 [2026-10-19 23:07:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.5467 [2026-10-19 23:07:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.4815 [2026-10-19 23:07:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.5909 [2026-10-19 23:07:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.8533 [2026-10-19 23:07:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.4869 [2026-10-19 23:07:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.7184 [2026-10-19 23:08:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.7756 [2026-10-19 23:08:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.3304 [2026-10-19 23:08:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.3095 [2026-10-19 23:08:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.7042 [2026-10-19 23:08:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.5154 [2026-10-19 23:08:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.0005 [2026-10-19 23:08:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.7804 [2026-10-19 23:08:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.6970 [2026-10-19 23:08:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.3710 [2026-10-19 23:08:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.0913 [2026-10-19 23:08:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.5861 [2026-10-19 23:08:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.7913 [2026-10-19 23:08:56][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,556 | MFU: 1.8% [2026-10-19 23:09:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.6325 [2026-10-19 23:09:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.7163 [2026-10-19 23:09:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.6995 [2026-10-19 23:09:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.5355 [2026-10-19 23:09:00][INFO][training.pretrain_engine] [Ph2|Step 112] loss=8.7157 | ppl=6098.10 | aux=0.02530 | lr=9.74e-05 | gnorm=1.234 | tok/s=29,274 | MFU=7.0% | step=143277ms | tokens=0.125B | mem=4.2/32.7GB | cost=$58.82/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 20:17 (141.1h) | Phase3: SKIPPED | SFT: Oct 26 01:02 (145.9h) | DPO (Final): Oct 26 01:36 (146.5h) [2026-10-19 23:09:00][INFO][training.pretrain_engine] Starting global step 113 (accumulating 32 micro-batches)... [2026-10-19 23:09:00][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 23:09:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.8435 [2026-10-19 23:09:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.5268 [2026-10-19 23:09:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.7539 [2026-10-19 23:09:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.3740 [2026-10-19 23:09:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.6396 [2026-10-19 23:09:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.3055 [2026-10-19 23:09:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.9146 [2026-10-19 23:09:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.6473 [2026-10-19 23:09:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.6708 [2026-10-19 23:09:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.1938 [2026-10-19 23:09:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.2208 [2026-10-19 23:09:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.5072 [2026-10-19 23:10:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.4396 [2026-10-19 23:10:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.6908 [2026-10-19 23:10:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.7261 [2026-10-19 23:10:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.5692 [2026-10-19 23:10:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.8221 [2026-10-19 23:10:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.5284 [2026-10-19 23:10:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.5433 [2026-10-19 23:10:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.1454 [2026-10-19 23:10:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.5056 [2026-10-19 23:10:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.3729 [2026-10-19 23:10:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.5728 [2026-10-19 23:10:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.4378 [2026-10-19 23:11:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.8781 [2026-10-19 23:11:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.8868 [2026-10-19 23:11:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.4842 [2026-10-19 23:11:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.6454 [2026-10-19 23:11:19][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,560 | MFU: 1.8% [2026-10-19 23:11:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.2928 [2026-10-19 23:11:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.2355 [2026-10-19 23:11:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.1638 [2026-10-19 23:11:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.9811 [2026-10-19 23:11:24][INFO][training.pretrain_engine] [Ph2|Step 113] loss=8.9182 | ppl=7466.44 | aux=0.02532 | lr=9.79e-05 | gnorm=1.062 | tok/s=29,275 | MFU=7.0% | step=143274ms | tokens=0.126B | mem=4.2/32.7GB | cost=$59.34/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 20:19 (141.1h) | Phase3: SKIPPED | SFT: Oct 26 01:04 (145.9h) | DPO (Final): Oct 26 01:38 (146.4h) [2026-10-19 23:11:24][INFO][training.pretrain_engine] Starting global step 114 (accumulating 32 micro-batches)... [2026-10-19 23:11:24][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 23:11:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.4089 [2026-10-19 23:11:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.3574 [2026-10-19 23:11:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.3645 [2026-10-19 23:11:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.1381 [2026-10-19 23:11:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.1579 [2026-10-19 23:11:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.5840 [2026-10-19 23:11:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.2182 [2026-10-19 23:12:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.4174 [2026-10-19 23:12:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.6439 [2026-10-19 23:12:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.0786 [2026-10-19 23:12:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.9372 [2026-10-19 23:12:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.5657 [2026-10-19 23:12:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.6062 [2026-10-19 23:12:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.4371 [2026-10-19 23:12:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.5184 [2026-10-19 23:12:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.3317 [2026-10-19 23:12:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.8325 [2026-10-19 23:12:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.5545 [2026-10-19 23:12:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.4498 [2026-10-19 23:12:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.4001 [2026-10-19 23:13:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.7282 [2026-10-19 23:13:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.0773 [2026-10-19 23:13:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.1863 [2026-10-19 23:13:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.5567 [2026-10-19 23:13:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.4526 [2026-10-19 23:13:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.8595 [2026-10-19 23:13:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.5999 [2026-10-19 23:13:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.7633 [2026-10-19 23:13:43][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,551 | MFU: 1.8% [2026-10-19 23:13:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.5477 [2026-10-19 23:13:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.0954 [2026-10-19 23:13:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.4406 [2026-10-19 23:13:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.1887 [2026-10-19 23:13:47][INFO][training.pretrain_engine] [Ph2|Step 114] loss=8.7411 | ppl=6254.70 | aux=0.02524 | lr=9.85e-05 | gnorm=1.000 | tok/s=29,274 | MFU=7.0% | step=143276ms | tokens=0.127B | mem=4.2/32.7GB | cost=$59.87/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 20:21 (141.1h) | Phase3: SKIPPED | SFT: Oct 26 01:06 (145.9h) | DPO (Final): Oct 26 01:40 (146.4h) [2026-10-19 23:13:47][INFO][training.pretrain_engine] Starting global step 115 (accumulating 32 micro-batches)... [2026-10-19 23:13:47][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 23:14:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.6795 [2026-10-19 23:14:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.4953 [2026-10-19 23:14:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.4387 [2026-10-19 23:14:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.4213 [2026-10-19 23:14:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.3854 [2026-10-19 23:14:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.1906 [2026-10-19 23:14:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.2956 [2026-10-19 23:14:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.4250 [2026-10-19 23:14:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.2374 [2026-10-19 23:14:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.5359 [2026-10-19 23:14:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.1525 [2026-10-19 23:14:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.4391 [2026-10-19 23:14:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.6327 [2026-10-19 23:14:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.8611 [2026-10-19 23:14:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.7371 [2026-10-19 23:14:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.9374 [2026-10-19 23:15:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.6361 [2026-10-19 23:15:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.0598 [2026-10-19 23:15:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.9450 [2026-10-19 23:15:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.9703 [2026-10-19 23:15:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.7400 [2026-10-19 23:15:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.8010 [2026-10-19 23:15:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.9132 [2026-10-19 23:15:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.9550 [2026-10-19 23:15:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.8193 [2026-10-19 23:15:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.9600 [2026-10-19 23:15:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.7888 [2026-10-19 23:15:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.9303 [2026-10-19 23:16:06][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,559 | MFU: 1.8% [2026-10-19 23:16:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.0332 [2026-10-19 23:16:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.6014 [2026-10-19 23:16:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.6381 [2026-10-19 23:16:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.4247 [2026-10-19 23:16:10][INFO][training.pretrain_engine] [Ph2|Step 115] loss=8.7634 | ppl=6395.95 | aux=0.02526 | lr=9.91e-05 | gnorm=0.930 | tok/s=29,274 | MFU=7.0% | step=143276ms | tokens=0.128B | mem=4.2/32.7GB | cost=$60.39/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 20:23 (141.1h) | Phase3: SKIPPED | SFT: Oct 26 01:07 (145.9h) | DPO (Final): Oct 26 01:42 (146.4h) [2026-10-19 23:16:10][INFO][training.pretrain_engine] Starting global step 116 (accumulating 32 micro-batches)... [2026-10-19 23:16:10][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 23:16:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.3095 [2026-10-19 23:16:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.3570 [2026-10-19 23:16:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.3888 [2026-10-19 23:16:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.4708 [2026-10-19 23:16:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.6458 [2026-10-19 23:16:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.7809 [2026-10-19 23:16:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.6397 [2026-10-19 23:16:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.6534 [2026-10-19 23:17:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.1910 [2026-10-19 23:17:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.4955 [2026-10-19 23:17:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.0559 [2026-10-19 23:17:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.5463 [2026-10-19 23:17:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.8388 [2026-10-19 23:17:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.3480 [2026-10-19 23:17:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.8701 [2026-10-19 23:17:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.3833 [2026-10-19 23:17:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.6310 [2026-10-19 23:17:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.6695 [2026-10-19 23:17:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.6067 [2026-10-19 23:17:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.9287 [2026-10-19 23:17:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.6673 [2026-10-19 23:17:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.3333 [2026-10-19 23:17:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.5741 [2026-10-19 23:17:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.4316 [2026-10-19 23:18:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.4578 [2026-10-19 23:18:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.8358 [2026-10-19 23:18:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.0937 [2026-10-19 23:18:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.3113 [2026-10-19 23:18:29][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,576 | MFU: 1.8% [2026-10-19 23:18:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.7073 [2026-10-19 23:18:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.6003 [2026-10-19 23:18:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.0231 [2026-10-19 23:18:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.2645 [2026-10-19 23:18:33][INFO][training.pretrain_engine] [Ph2|Step 116] loss=8.6550 | ppl=5738.91 | aux=0.02527 | lr=9.96e-05 | gnorm=0.973 | tok/s=29,275 | MFU=7.0% | step=143272ms | tokens=0.129B | mem=4.2/32.7GB | cost=$60.91/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 20:24 (141.1h) | Phase3: SKIPPED | SFT: Oct 26 01:09 (145.8h) | DPO (Final): Oct 26 01:43 (146.4h) [2026-10-19 23:18:33][INFO][training.pretrain_engine] Starting global step 117 (accumulating 32 micro-batches)... [2026-10-19 23:18:33][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 23:18:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.3775 [2026-10-19 23:18:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.2649 [2026-10-19 23:18:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.5474 [2026-10-19 23:18:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.3883 [2026-10-19 23:19:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.5206 [2026-10-19 23:19:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.7132 [2026-10-19 23:19:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.4879 [2026-10-19 23:19:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.3444 [2026-10-19 23:19:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.2306 [2026-10-19 23:19:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.1590 [2026-10-19 23:19:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.3826 [2026-10-19 23:19:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.3589 [2026-10-19 23:19:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.4783 [2026-10-19 23:19:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.5241 [2026-10-19 23:19:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.5911 [2026-10-19 23:19:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.5260 [2026-10-19 23:20:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.8914 [2026-10-19 23:20:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.9505 [2026-10-19 23:20:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.8584 [2026-10-19 23:20:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.2880 [2026-10-19 23:20:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.0709 [2026-10-19 23:20:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.6931 [2026-10-19 23:20:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.6630 [2026-10-19 23:20:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.0701 [2026-10-19 23:20:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.7210 [2026-10-19 23:20:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.8722 [2026-10-19 23:20:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.7868 [2026-10-19 23:20:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.0944 [2026-10-19 23:20:52][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,549 | MFU: 1.8% [2026-10-19 23:20:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.1753 [2026-10-19 23:20:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.3664 [2026-10-19 23:20:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.7831 [2026-10-19 23:20:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.4811 [2026-10-19 23:20:57][INFO][training.pretrain_engine] [Ph2|Step 117] loss=8.7117 | ppl=6073.58 | aux=0.02523 | lr=1.00e-04 | gnorm=0.926 | tok/s=29,275 | MFU=7.0% | step=143272ms | tokens=0.130B | mem=4.2/32.7GB | cost=$61.44/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 20:26 (141.1h) | Phase3: SKIPPED | SFT: Oct 26 01:11 (145.8h) | DPO (Final): Oct 26 01:45 (146.4h) [2026-10-19 23:21:00][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step117 | Tokens: 0.130B | Spent: $61.45 | Remaining: $338.55 [2026-10-19 23:21:00][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step117 [2026-10-19 23:21:00][INFO][training.pretrain_engine] Starting global step 118 (accumulating 32 micro-batches)... [2026-10-19 23:21:00][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 23:21:00][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-19 23:21:00][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-19 23:21:00][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-19 23:21:00][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step117 to Phase-Technologies/forge-500m-pretrain (path: phase2_step117)... [2026-10-19 23:21:00][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-19 23:21:10][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-19 23:21:10][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step117 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-19 23:21:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.2070 [2026-10-19 23:21:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.2912 [2026-10-19 23:21:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.2495 [2026-10-19 23:21:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.3271 [2026-10-19 23:21:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.3861 [2026-10-19 23:21:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.3180 [2026-10-19 23:21:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.1205 [2026-10-19 23:21:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.6608 [2026-10-19 23:21:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.6595 [2026-10-19 23:21:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.4209 [2026-10-19 23:21:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.7974 [2026-10-19 23:21:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.4139 [2026-10-19 23:22:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.4839 [2026-10-19 23:22:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.6886 [2026-10-19 23:22:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.1601 [2026-10-19 23:22:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.8793 [2026-10-19 23:22:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.3911 [2026-10-19 23:22:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.7481 [2026-10-19 23:22:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.4633 [2026-10-19 23:22:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.5998 [2026-10-19 23:22:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.9070 [2026-10-19 23:22:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.1838 [2026-10-19 23:22:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.8776 [2026-10-19 23:22:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.0864 [2026-10-19 23:23:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.6739 [2026-10-19 23:23:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.6660 [2026-10-19 23:23:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.2988 [2026-10-19 23:23:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.9863 [2026-10-19 23:23:19][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,570 | MFU: 1.8% [2026-10-19 23:23:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.7432 [2026-10-19 23:23:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.1020 [2026-10-19 23:23:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.9122 [2026-10-19 23:23:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.1064 [2026-10-19 23:23:23][INFO][training.pretrain_engine] [Ph2|Step 118] loss=8.5792 | ppl=5319.98 | aux=0.02525 | lr=1.01e-04 | gnorm=0.980 | tok/s=29,276 | MFU=7.0% | step=143268ms | tokens=0.131B | mem=4.2/32.7GB | cost=$61.97/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 20:28 (141.1h) | Phase3: SKIPPED | SFT: Oct 26 01:12 (145.8h) | DPO (Final): Oct 26 01:46 (146.4h) [2026-10-19 23:23:23][INFO][training.pretrain_engine] Starting global step 119 (accumulating 32 micro-batches)... [2026-10-19 23:23:23][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 23:23:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.2191 [2026-10-19 23:23:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.6833 [2026-10-19 23:23:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.7970 [2026-10-19 23:23:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.6142 [2026-10-19 23:23:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.0105 [2026-10-19 23:23:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.1323 [2026-10-19 23:23:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.2993 [2026-10-19 23:23:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.3290 [2026-10-19 23:24:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.4463 [2026-10-19 23:24:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.5141 [2026-10-19 23:24:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.3304 [2026-10-19 23:24:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.4829 [2026-10-19 23:24:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.3109 [2026-10-19 23:24:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.1606 [2026-10-19 23:24:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.2537 [2026-10-19 23:24:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.4776 [2026-10-19 23:24:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.3463 [2026-10-19 23:24:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.2878 [2026-10-19 23:24:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.2649 [2026-10-19 23:24:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.2121 [2026-10-19 23:25:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.2414 [2026-10-19 23:25:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.1048 [2026-10-19 23:25:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.3310 [2026-10-19 23:25:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.3720 [2026-10-19 23:25:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.3909 [2026-10-19 23:25:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.4212 [2026-10-19 23:25:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.8350 [2026-10-19 23:25:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.3899 [2026-10-19 23:25:41][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,584 | MFU: 1.8% [2026-10-19 23:25:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.2390 [2026-10-19 23:25:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.9177 [2026-10-19 23:25:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.2343 [2026-10-19 23:25:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.9360 [2026-10-19 23:25:46][INFO][training.pretrain_engine] [Ph2|Step 119] loss=8.6258 | ppl=5573.86 | aux=0.02526 | lr=1.01e-04 | gnorm=0.961 | tok/s=29,276 | MFU=7.0% | step=143266ms | tokens=0.132B | mem=4.2/32.7GB | cost=$62.50/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 20:29 (141.1h) | Phase3: SKIPPED | SFT: Oct 26 01:14 (145.8h) | DPO (Final): Oct 26 01:48 (146.4h) [2026-10-19 23:25:46][INFO][training.pretrain_engine] Starting global step 120 (accumulating 32 micro-batches)... [2026-10-19 23:25:46][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 23:26:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.3373 [2026-10-19 23:26:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.1894 [2026-10-19 23:26:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.2317 [2026-10-19 23:26:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.4006 [2026-10-19 23:26:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.3530 [2026-10-19 23:26:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.2788 [2026-10-19 23:26:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.2333 [2026-10-19 23:26:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.6200 [2026-10-19 23:26:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.6228 [2026-10-19 23:26:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.8554 [2026-10-19 23:26:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.8071 [2026-10-19 23:26:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.9580 [2026-10-19 23:26:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.7571 [2026-10-19 23:26:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.4824 [2026-10-19 23:26:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.5203 [2026-10-19 23:26:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.5504 [2026-10-19 23:27:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.8083 [2026-10-19 23:27:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.0981 [2026-10-19 23:27:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.9240 [2026-10-19 23:27:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.2398 [2026-10-19 23:27:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.4021 [2026-10-19 23:27:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.3997 [2026-10-19 23:27:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.1931 [2026-10-19 23:27:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.5009 [2026-10-19 23:27:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.1428 [2026-10-19 23:27:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.1291 [2026-10-19 23:27:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.9242 [2026-10-19 23:27:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.9164 [2026-10-19 23:28:04][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,614 | MFU: 1.8% [2026-10-19 23:28:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.0511 [2026-10-19 23:28:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.0641 [2026-10-19 23:28:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.9766 [2026-10-19 23:28:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.0816 [2026-10-19 23:28:09][INFO][training.pretrain_engine] [Ph2|Step 120] loss=8.5732 | ppl=5287.79 | aux=0.02523 | lr=1.02e-04 | gnorm=1.047 | tok/s=29,277 | MFU=7.0% | step=143261ms | tokens=0.133B | mem=4.2/32.7GB | cost=$63.02/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 20:31 (141.1h) | Phase3: SKIPPED | SFT: Oct 26 01:15 (145.8h) | DPO (Final): Oct 26 01:50 (146.4h) [2026-10-19 23:28:09][INFO][training.pretrain_engine] Starting global step 121 (accumulating 32 micro-batches)... [2026-10-19 23:28:09][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 23:28:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.1511 [2026-10-19 23:28:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.0996 [2026-10-19 23:28:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.2686 [2026-10-19 23:28:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.3331 [2026-10-19 23:28:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.5451 [2026-10-19 23:28:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.6273 [2026-10-19 23:28:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.5230 [2026-10-19 23:28:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.6749 [2026-10-19 23:29:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.2251 [2026-10-19 23:29:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.4278 [2026-10-19 23:29:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.3153 [2026-10-19 23:29:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.3281 [2026-10-19 23:29:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.2695 [2026-10-19 23:29:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.3612 [2026-10-19 23:29:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.3300 [2026-10-19 23:29:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.1866 [2026-10-19 23:29:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.2832 [2026-10-19 23:29:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.1442 [2026-10-19 23:29:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.3045 [2026-10-19 23:29:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.7464 [2026-10-19 23:29:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.5749 [2026-10-19 23:29:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.5411 [2026-10-19 23:29:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.6154 [2026-10-19 23:29:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.5201 [2026-10-19 23:30:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.5815 [2026-10-19 23:30:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.4053 [2026-10-19 23:30:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.6087 [2026-10-19 23:30:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.4853 [2026-10-19 23:30:27][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,580 | MFU: 1.8% [2026-10-19 23:30:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.1136 [2026-10-19 23:30:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.9930 [2026-10-19 23:30:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.2865 [2026-10-19 23:30:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.5127 [2026-10-19 23:30:32][INFO][training.pretrain_engine] [Ph2|Step 121] loss=8.4635 | ppl=4738.77 | aux=0.02519 | lr=1.02e-04 | gnorm=1.109 | tok/s=29,279 | MFU=7.0% | step=143254ms | tokens=0.134B | mem=4.2/32.7GB | cost=$63.54/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 20:32 (141.0h) | Phase3: SKIPPED | SFT: Oct 26 01:17 (145.8h) | DPO (Final): Oct 26 01:51 (146.3h) [2026-10-19 23:30:32][INFO][training.pretrain_engine] Starting global step 122 (accumulating 32 micro-batches)... [2026-10-19 23:30:32][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 23:30:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.1332 [2026-10-19 23:30:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.5123 [2026-10-19 23:30:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.4070 [2026-10-19 23:30:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.6007 [2026-10-19 23:31:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.2765 [2026-10-19 23:31:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.6021 [2026-10-19 23:31:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.5293 [2026-10-19 23:31:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.3141 [2026-10-19 23:31:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.3136 [2026-10-19 23:31:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.3294 [2026-10-19 23:31:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.0851 [2026-10-19 23:31:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.3331 [2026-10-19 23:31:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.8936 [2026-10-19 23:31:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.2689 [2026-10-19 23:31:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.4415 [2026-10-19 23:31:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.4303 [2026-10-19 23:32:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.2832 [2026-10-19 23:32:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.3336 [2026-10-19 23:32:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.5998 [2026-10-19 23:32:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.2247 [2026-10-19 23:32:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.8334 [2026-10-19 23:32:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.7898 [2026-10-19 23:32:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.7077 [2026-10-19 23:32:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.6398 [2026-10-19 23:32:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.4871 [2026-10-19 23:32:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.8835 [2026-10-19 23:32:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.3203 [2026-10-19 23:32:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.6731 [2026-10-19 23:32:50][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,553 | MFU: 1.8% [2026-10-19 23:32:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.4921 [2026-10-19 23:32:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.3308 [2026-10-19 23:32:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.2456 [2026-10-19 23:32:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.2823 [2026-10-19 23:32:55][INFO][training.pretrain_engine] [Ph2|Step 122] loss=8.5022 | ppl=4925.48 | aux=0.02523 | lr=1.03e-04 | gnorm=1.086 | tok/s=29,279 | MFU=7.0% | step=143255ms | tokens=0.135B | mem=4.2/32.7GB | cost=$64.06/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 20:34 (141.0h) | Phase3: SKIPPED | SFT: Oct 26 01:19 (145.8h) | DPO (Final): Oct 26 01:53 (146.3h) [2026-10-19 23:32:55][INFO][training.pretrain_engine] Starting global step 123 (accumulating 32 micro-batches)... [2026-10-19 23:32:55][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 23:33:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.6465 [2026-10-19 23:33:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.1963 [2026-10-19 23:33:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.3476 [2026-10-19 23:33:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.6600 [2026-10-19 23:33:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.6550 [2026-10-19 23:33:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.9178 [2026-10-19 23:33:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.7954 [2026-10-19 23:33:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.7217 [2026-10-19 23:33:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.9612 [2026-10-19 23:33:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.0880 [2026-10-19 23:33:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.9122 [2026-10-19 23:33:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.0698 [2026-10-19 23:34:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.3467 [2026-10-19 23:34:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.6535 [2026-10-19 23:34:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.1814 [2026-10-19 23:34:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.2281 [2026-10-19 23:34:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.2870 [2026-10-19 23:34:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.1486 [2026-10-19 23:34:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.1405 [2026-10-19 23:34:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.2318 [2026-10-19 23:34:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.7781 [2026-10-19 23:34:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.1775 [2026-10-19 23:34:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.6165 [2026-10-19 23:34:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.6188 [2026-10-19 23:34:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.2924 [2026-10-19 23:35:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.0893 [2026-10-19 23:35:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.5590 [2026-10-19 23:35:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.4025 [2026-10-19 23:35:14][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,546 | MFU: 1.8% [2026-10-19 23:35:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.6987 [2026-10-19 23:35:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.7916 [2026-10-19 23:35:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.9002 [2026-10-19 23:35:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.0512 [2026-10-19 23:35:18][INFO][training.pretrain_engine] [Ph2|Step 123] loss=8.6564 | ppl=5746.53 | aux=0.02524 | lr=1.04e-04 | gnorm=1.102 | tok/s=29,279 | MFU=7.0% | step=143253ms | tokens=0.136B | mem=4.2/32.7GB | cost=$64.59/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 20:36 (141.0h) | Phase3: SKIPPED | SFT: Oct 26 01:20 (145.8h) | DPO (Final): Oct 26 01:55 (146.3h) [2026-10-19 23:35:18][INFO][training.pretrain_engine] Starting global step 124 (accumulating 32 micro-batches)... [2026-10-19 23:35:18][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 23:35:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.3508 [2026-10-19 23:35:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.6757 [2026-10-19 23:35:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.1285 [2026-10-19 23:35:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.1995 [2026-10-19 23:35:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.2192 [2026-10-19 23:35:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.1984 [2026-10-19 23:35:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.0935 [2026-10-19 23:35:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.1507 [2026-10-19 23:36:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.2420 [2026-10-19 23:36:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.3006 [2026-10-19 23:36:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.1252 [2026-10-19 23:36:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.0944 [2026-10-19 23:36:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.2592 [2026-10-19 23:36:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.1445 [2026-10-19 23:36:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.2752 [2026-10-19 23:36:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.3369 [2026-10-19 23:36:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.2836 [2026-10-19 23:36:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.4639 [2026-10-19 23:36:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.6345 [2026-10-19 23:36:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.6091 [2026-10-19 23:37:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.7894 [2026-10-19 23:37:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.3305 [2026-10-19 23:37:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.7609 [2026-10-19 23:37:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=9.1803 [2026-10-19 23:37:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.6347 [2026-10-19 23:37:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.5837 [2026-10-19 23:37:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.2461 [2026-10-19 23:37:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.3281 [2026-10-19 23:37:37][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,547 | MFU: 1.8% [2026-10-19 23:37:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.3999 [2026-10-19 23:37:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.2462 [2026-10-19 23:37:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.3630 [2026-10-19 23:37:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.6442 [2026-10-19 23:37:42][INFO][training.pretrain_engine] [Ph2|Step 124] loss=8.4961 | ppl=4895.44 | aux=0.02531 | lr=1.04e-04 | gnorm=1.227 | tok/s=29,278 | MFU=7.0% | step=143256ms | tokens=0.137B | mem=4.2/32.7GB | cost=$65.11/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 20:38 (141.0h) | Phase3: SKIPPED | SFT: Oct 26 01:22 (145.8h) | DPO (Final): Oct 26 01:57 (146.3h) [2026-10-19 23:37:42][INFO][training.pretrain_engine] Starting global step 125 (accumulating 32 micro-batches)... [2026-10-19 23:37:42][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 23:38:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.1771 [2026-10-19 23:38:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.8251 [2026-10-19 23:38:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.3555 [2026-10-19 23:38:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.4176 [2026-10-19 23:38:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.0612 [2026-10-19 23:38:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.3715 [2026-10-19 23:38:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.3817 [2026-10-19 23:38:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.5202 [2026-10-19 23:38:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.3338 [2026-10-19 23:38:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.7274 [2026-10-19 23:38:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.8699 [2026-10-19 23:38:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.2265 [2026-10-19 23:38:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.1321 [2026-10-19 23:38:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.7751 [2026-10-19 23:38:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.3566 [2026-10-19 23:38:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.2624 [2026-10-19 23:39:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.7749 [2026-10-19 23:39:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.8141 [2026-10-19 23:39:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.8734 [2026-10-19 23:39:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.5090 [2026-10-19 23:39:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.3069 [2026-10-19 23:39:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.9257 [2026-10-19 23:39:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.5836 [2026-10-19 23:39:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.4486 [2026-10-19 23:39:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.4636 [2026-10-19 23:39:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.1972 [2026-10-19 23:39:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.8102 [2026-10-19 23:39:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.3930 [2026-10-19 23:40:01][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,543 | MFU: 1.8% [2026-10-19 23:40:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.1688 [2026-10-19 23:40:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.1303 [2026-10-19 23:40:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.0901 [2026-10-19 23:40:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.3984 [2026-10-19 23:40:05][INFO][training.pretrain_engine] [Ph2|Step 125] loss=8.4833 | ppl=4833.14 | aux=0.02522 | lr=1.05e-04 | gnorm=1.039 | tok/s=29,279 | MFU=7.0% | step=143253ms | tokens=0.138B | mem=4.2/32.7GB | cost=$65.64/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 20:39 (141.0h) | Phase3: SKIPPED | SFT: Oct 26 01:24 (145.7h) | DPO (Final): Oct 26 01:58 (146.3h) [2026-10-19 23:40:05][INFO][training.pretrain_engine] Starting global step 126 (accumulating 32 micro-batches)... [2026-10-19 23:40:05][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 23:40:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.5697 [2026-10-19 23:40:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.5290 [2026-10-19 23:40:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.5854 [2026-10-19 23:40:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.8042 [2026-10-19 23:40:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.1967 [2026-10-19 23:40:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.4405 [2026-10-19 23:40:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.3319 [2026-10-19 23:40:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.7726 [2026-10-19 23:40:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.1912 [2026-10-19 23:40:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.0758 [2026-10-19 23:40:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.1218 [2026-10-19 23:40:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.1260 [2026-10-19 23:41:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.3578 [2026-10-19 23:41:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.2482 [2026-10-19 23:41:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.2412 [2026-10-19 23:41:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.5592 [2026-10-19 23:41:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.4362 [2026-10-19 23:41:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.4576 [2026-10-19 23:41:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.6125 [2026-10-19 23:41:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.4562 [2026-10-19 23:41:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.1542 [2026-10-19 23:41:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.3256 [2026-10-19 23:41:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.1341 [2026-10-19 23:41:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.0913 [2026-10-19 23:42:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.1699 [2026-10-19 23:42:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.4294 [2026-10-19 23:42:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.3742 [2026-10-19 23:42:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.3327 [2026-10-19 23:42:24][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,555 | MFU: 1.8% [2026-10-19 23:42:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.2701 [2026-10-19 23:42:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.1314 [2026-10-19 23:42:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.2409 [2026-10-19 23:42:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.2902 [2026-10-19 23:42:29][INFO][training.pretrain_engine] [Ph2|Step 126] loss=8.4653 | ppl=4747.02 | aux=0.02524 | lr=1.05e-04 | gnorm=0.965 | tok/s=29,280 | MFU=7.0% | step=143250ms | tokens=0.139B | mem=4.2/32.7GB | cost=$66.16/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 20:41 (141.0h) | Phase3: SKIPPED | SFT: Oct 26 01:26 (145.7h) | DPO (Final): Oct 26 02:00 (146.3h) [2026-10-19 23:42:32][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step126 | Tokens: 0.139B | Spent: $66.17 | Remaining: $333.83 [2026-10-19 23:42:32][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step126 [2026-10-19 23:42:32][INFO][training.pretrain_engine] Starting global step 127 (accumulating 32 micro-batches)... [2026-10-19 23:42:32][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 23:42:32][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-19 23:42:32][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-19 23:42:32][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-19 23:42:32][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step126 to Phase-Technologies/forge-500m-pretrain (path: phase2_step126)... [2026-10-19 23:42:33][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-19 23:42:42][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-19 23:42:42][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step126 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-19 23:42:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.6877 [2026-10-19 23:42:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.1721 [2026-10-19 23:42:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.2903 [2026-10-19 23:42:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.3037 [2026-10-19 23:43:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.0841 [2026-10-19 23:43:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.1853 [2026-10-19 23:43:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.4304 [2026-10-19 23:43:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.8369 [2026-10-19 23:43:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.2366 [2026-10-19 23:43:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.3538 [2026-10-19 23:43:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.2614 [2026-10-19 23:43:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.1759 [2026-10-19 23:43:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.2600 [2026-10-19 23:43:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.6601 [2026-10-19 23:43:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.2372 [2026-10-19 23:43:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.7522 [2026-10-19 23:43:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.4293 [2026-10-19 23:43:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.5615 [2026-10-19 23:43:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.3848 [2026-10-19 23:44:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.5689 [2026-10-19 23:44:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.2843 [2026-10-19 23:44:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.7835 [2026-10-19 23:44:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.2444 [2026-10-19 23:44:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.6970 [2026-10-19 23:44:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.2871 [2026-10-19 23:44:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1499 [2026-10-19 23:44:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.2727 [2026-10-19 23:44:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.4137 [2026-10-19 23:44:51][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,578 | MFU: 1.8% [2026-10-19 23:44:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.0223 [2026-10-19 23:44:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.1263 [2026-10-19 23:44:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.2885 [2026-10-19 23:44:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.9706 [2026-10-19 23:44:55][INFO][training.pretrain_engine] [Ph2|Step 127] loss=8.5114 | ppl=4971.09 | aux=0.02524 | lr=1.06e-04 | gnorm=1.117 | tok/s=29,280 | MFU=7.0% | step=143246ms | tokens=0.141B | mem=4.2/32.7GB | cost=$66.70/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 20:43 (141.0h) | Phase3: SKIPPED | SFT: Oct 26 01:27 (145.7h) | DPO (Final): Oct 26 02:01 (146.3h) [2026-10-19 23:44:55][INFO][training.pretrain_engine] Starting global step 128 (accumulating 32 micro-batches)... [2026-10-19 23:44:55][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 23:45:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.1718 [2026-10-19 23:45:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.6589 [2026-10-19 23:45:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.2341 [2026-10-19 23:45:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.3879 [2026-10-19 23:45:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.2015 [2026-10-19 23:45:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.2187 [2026-10-19 23:45:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.3689 [2026-10-19 23:45:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.5043 [2026-10-19 23:45:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.2804 [2026-10-19 23:45:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.2680 [2026-10-19 23:45:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.1036 [2026-10-19 23:45:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.2774 [2026-10-19 23:46:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.5534 [2026-10-19 23:46:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.5502 [2026-10-19 23:46:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.7972 [2026-10-19 23:46:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.4843 [2026-10-19 23:46:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.6981 [2026-10-19 23:46:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.8289 [2026-10-19 23:46:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.5304 [2026-10-19 23:46:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.5864 [2026-10-19 23:46:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.7769 [2026-10-19 23:46:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.3679 [2026-10-19 23:46:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.5078 [2026-10-19 23:46:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.1932 [2026-10-19 23:47:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.9575 [2026-10-19 23:47:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.1106 [2026-10-19 23:47:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.0890 [2026-10-19 23:47:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.1153 [2026-10-19 23:47:14][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,563 | MFU: 1.8% [2026-10-19 23:47:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.3332 [2026-10-19 23:47:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.3301 [2026-10-19 23:47:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.9683 [2026-10-19 23:47:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.9628 [2026-10-19 23:47:18][INFO][training.pretrain_engine] [Ph2|Step 128] loss=8.3824 | ppl=4369.41 | aux=0.02524 | lr=1.06e-04 | gnorm=1.008 | tok/s=29,281 | MFU=7.0% | step=143245ms | tokens=0.142B | mem=4.2/32.7GB | cost=$67.22/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 20:44 (141.0h) | Phase3: SKIPPED | SFT: Oct 26 01:29 (145.7h) | DPO (Final): Oct 26 02:03 (146.3h) [2026-10-19 23:47:18][INFO][training.pretrain_engine] Starting global step 129 (accumulating 32 micro-batches)... [2026-10-19 23:47:18][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 23:47:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.2065 [2026-10-19 23:47:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.1493 [2026-10-19 23:47:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.0682 [2026-10-19 23:47:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.9101 [2026-10-19 23:47:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.3738 [2026-10-19 23:47:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.3461 [2026-10-19 23:47:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.2663 [2026-10-19 23:47:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.0577 [2026-10-19 23:48:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.0325 [2026-10-19 23:48:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.8789 [2026-10-19 23:48:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.8978 [2026-10-19 23:48:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.3047 [2026-10-19 23:48:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.7577 [2026-10-19 23:48:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.8665 [2026-10-19 23:48:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.6310 [2026-10-19 23:48:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.0839 [2026-10-19 23:48:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.3370 [2026-10-19 23:48:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.1430 [2026-10-19 23:48:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.4895 [2026-10-19 23:48:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.2628 [2026-10-19 23:49:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.1721 [2026-10-19 23:49:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.1852 [2026-10-19 23:49:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.5301 [2026-10-19 23:49:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.5663 [2026-10-19 23:49:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.3576 [2026-10-19 23:49:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.5164 [2026-10-19 23:49:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.3225 [2026-10-19 23:49:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.3251 [2026-10-19 23:49:37][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,552 | MFU: 1.8% [2026-10-19 23:49:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.0835 [2026-10-19 23:49:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.2630 [2026-10-19 23:49:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.4607 [2026-10-19 23:49:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.2460 [2026-10-19 23:49:42][INFO][training.pretrain_engine] [Ph2|Step 129] loss=8.4101 | ppl=4492.16 | aux=0.02518 | lr=1.07e-04 | gnorm=1.008 | tok/s=29,280 | MFU=7.0% | step=143247ms | tokens=0.143B | mem=4.2/32.7GB | cost=$67.74/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 20:46 (140.9h) | Phase3: SKIPPED | SFT: Oct 26 01:31 (145.7h) | DPO (Final): Oct 26 02:05 (146.3h) [2026-10-19 23:49:42][INFO][training.pretrain_engine] Starting global step 130 (accumulating 32 micro-batches)... [2026-10-19 23:49:42][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 23:49:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.9917 [2026-10-19 23:49:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.4679 [2026-10-19 23:49:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.1389 [2026-10-19 23:50:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.4095 [2026-10-19 23:50:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.3879 [2026-10-19 23:50:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.1273 [2026-10-19 23:50:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.4190 [2026-10-19 23:50:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.2823 [2026-10-19 23:50:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.3380 [2026-10-19 23:50:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.1596 [2026-10-19 23:50:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.8543 [2026-10-19 23:50:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.4152 [2026-10-19 23:50:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.7808 [2026-10-19 23:50:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.6146 [2026-10-19 23:50:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.4872 [2026-10-19 23:50:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.7259 [2026-10-19 23:51:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.4241 [2026-10-19 23:51:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.6159 [2026-10-19 23:51:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.1803 [2026-10-19 23:51:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.5787 [2026-10-19 23:51:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.3110 [2026-10-19 23:51:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.1981 [2026-10-19 23:51:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.3001 [2026-10-19 23:51:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.2804 [2026-10-19 23:51:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.1992 [2026-10-19 23:51:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.3945 [2026-10-19 23:51:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.3588 [2026-10-19 23:51:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.3170 [2026-10-19 23:52:00][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,587 | MFU: 1.8% [2026-10-19 23:52:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.6262 [2026-10-19 23:52:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.7213 [2026-10-19 23:52:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.9038 [2026-10-19 23:52:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.3060 [2026-10-19 23:52:05][INFO][training.pretrain_engine] [Ph2|Step 130] loss=8.3135 | ppl=4078.44 | aux=0.02520 | lr=1.08e-04 | gnorm=0.848 | tok/s=29,282 | MFU=7.0% | step=143239ms | tokens=0.144B | mem=4.2/32.7GB | cost=$68.27/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 20:48 (140.9h) | Phase3: SKIPPED | SFT: Oct 26 01:32 (145.7h) | DPO (Final): Oct 26 02:06 (146.2h) [2026-10-19 23:52:05][INFO][training.pretrain_engine] Starting global step 131 (accumulating 32 micro-batches)... [2026-10-19 23:52:05][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 23:52:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.2331 [2026-10-19 23:52:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.8157 [2026-10-19 23:52:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.5900 [2026-10-19 23:52:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.7161 [2026-10-19 23:52:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.9709 [2026-10-19 23:52:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.8546 [2026-10-19 23:52:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.1450 [2026-10-19 23:52:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.1051 [2026-10-19 23:52:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.3917 [2026-10-19 23:52:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.2062 [2026-10-19 23:52:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.0331 [2026-10-19 23:52:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.3183 [2026-10-19 23:53:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.1449 [2026-10-19 23:53:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.3651 [2026-10-19 23:53:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.2240 [2026-10-19 23:53:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.1708 [2026-10-19 23:53:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.1287 [2026-10-19 23:53:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.1118 [2026-10-19 23:53:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.2091 [2026-10-19 23:53:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.0658 [2026-10-19 23:53:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.2333 [2026-10-19 23:53:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.0845 [2026-10-19 23:53:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.3140 [2026-10-19 23:53:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.1401 [2026-10-19 23:54:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.2935 [2026-10-19 23:54:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.8471 [2026-10-19 23:54:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.9175 [2026-10-19 23:54:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.8188 [2026-10-19 23:54:23][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,565 | MFU: 1.8% [2026-10-19 23:54:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.0596 [2026-10-19 23:54:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.2164 [2026-10-19 23:54:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.1816 [2026-10-19 23:54:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.4043 [2026-10-19 23:54:28][INFO][training.pretrain_engine] [Ph2|Step 131] loss=8.2749 | ppl=3924.18 | aux=0.02520 | lr=1.08e-04 | gnorm=0.949 | tok/s=29,282 | MFU=7.0% | step=143237ms | tokens=0.145B | mem=4.2/32.7GB | cost=$68.79/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 20:49 (140.9h) | Phase3: SKIPPED | SFT: Oct 26 01:34 (145.7h) | DPO (Final): Oct 26 02:08 (146.2h) [2026-10-19 23:54:28][INFO][training.pretrain_engine] Starting global step 132 (accumulating 32 micro-batches)... [2026-10-19 23:54:28][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 23:54:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.2439 [2026-10-19 23:54:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.1286 [2026-10-19 23:54:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.3894 [2026-10-19 23:54:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.0366 [2026-10-19 23:55:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.9368 [2026-10-19 23:55:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.1819 [2026-10-19 23:55:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.1772 [2026-10-19 23:55:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.9231 [2026-10-19 23:55:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.1746 [2026-10-19 23:55:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.0032 [2026-10-19 23:55:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.6763 [2026-10-19 23:55:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.6247 [2026-10-19 23:55:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.1372 [2026-10-19 23:55:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.1728 [2026-10-19 23:55:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.1794 [2026-10-19 23:55:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.2398 [2026-10-19 23:55:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.3159 [2026-10-19 23:55:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.0682 [2026-10-19 23:55:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.0929 [2026-10-19 23:55:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.0515 [2026-10-19 23:56:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.4422 [2026-10-19 23:56:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.6673 [2026-10-19 23:56:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.6284 [2026-10-19 23:56:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.4549 [2026-10-19 23:56:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.1474 [2026-10-19 23:56:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.3265 [2026-10-19 23:56:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.4318 [2026-10-19 23:56:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.4262 [2026-10-19 23:56:46][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,556 | MFU: 1.8% [2026-10-19 23:56:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.0164 [2026-10-19 23:56:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.0854 [2026-10-19 23:56:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.1852 [2026-10-19 23:56:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.2386 [2026-10-19 23:56:51][INFO][training.pretrain_engine] [Ph2|Step 132] loss=8.2717 | ppl=3911.51 | aux=0.02523 | lr=1.09e-04 | gnorm=0.918 | tok/s=29,282 | MFU=7.0% | step=143237ms | tokens=0.146B | mem=4.2/32.7GB | cost=$69.31/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 20:51 (140.9h) | Phase3: SKIPPED | SFT: Oct 26 01:36 (145.7h) | DPO (Final): Oct 26 02:10 (146.2h) [2026-10-19 23:56:51][INFO][training.pretrain_engine] Starting global step 133 (accumulating 32 micro-batches)... [2026-10-19 23:56:51][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 23:57:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.7366 [2026-10-19 23:57:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.6779 [2026-10-19 23:57:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.9019 [2026-10-19 23:57:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.4651 [2026-10-19 23:57:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.7149 [2026-10-19 23:57:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.0590 [2026-10-19 23:57:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.5101 [2026-10-19 23:57:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.3566 [2026-10-19 23:57:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.1197 [2026-10-19 23:57:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.2374 [2026-10-19 23:57:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.3632 [2026-10-19 23:57:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.3393 [2026-10-19 23:58:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.7846 [2026-10-19 23:58:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.6007 [2026-10-19 23:58:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.6714 [2026-10-19 23:58:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.4268 [2026-10-19 23:58:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.3070 [2026-10-19 23:58:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.1363 [2026-10-19 23:58:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.1215 [2026-10-19 23:58:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.1442 [2026-10-19 23:58:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.2574 [2026-10-19 23:58:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.3100 [2026-10-19 23:58:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.2269 [2026-10-19 23:58:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.1416 [2026-10-19 23:58:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.3714 [2026-10-19 23:58:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.4416 [2026-10-19 23:58:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.1000 [2026-10-19 23:58:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.3949 [2026-10-19 23:59:10][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,547 | MFU: 1.8% [2026-10-19 23:59:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.6909 [2026-10-19 23:59:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.4050 [2026-10-19 23:59:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.8203 [2026-10-19 23:59:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.0444 [2026-10-19 23:59:14][INFO][training.pretrain_engine] [Ph2|Step 133] loss=8.3386 | ppl=4182.24 | aux=0.02522 | lr=1.09e-04 | gnorm=0.965 | tok/s=29,283 | MFU=7.0% | step=143235ms | tokens=0.147B | mem=4.2/32.7GB | cost=$69.84/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 20:53 (140.9h) | Phase3: SKIPPED | SFT: Oct 26 01:37 (145.6h) | DPO (Final): Oct 26 02:11 (146.2h) [2026-10-19 23:59:14][INFO][training.pretrain_engine] Starting global step 134 (accumulating 32 micro-batches)... [2026-10-19 23:59:14][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-19 23:59:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.8910 [2026-10-19 23:59:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.9054 [2026-10-19 23:59:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.5018 [2026-10-19 23:59:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.8752 [2026-10-19 23:59:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.1188 [2026-10-19 23:59:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.2096 [2026-10-19 23:59:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.1187 [2026-10-19 23:59:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.9732 [2026-10-20 00:00:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.2382 [2026-10-20 00:00:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.0494 [2026-10-20 00:00:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.2556 [2026-10-20 00:00:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.1669 [2026-10-20 00:00:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.3285 [2026-10-20 00:00:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.1696 [2026-10-20 00:00:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.6211 [2026-10-20 00:00:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.7678 [2026-10-20 00:00:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.9447 [2026-10-20 00:00:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.0586 [2026-10-20 00:00:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.0146 [2026-10-20 00:00:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.9291 [2026-10-20 00:01:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.0702 [2026-10-20 00:01:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.1550 [2026-10-20 00:01:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.0670 [2026-10-20 00:01:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.9124 [2026-10-20 00:01:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.1779 [2026-10-20 00:01:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.9341 [2026-10-20 00:01:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.8433 [2026-10-20 00:01:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.7408 [2026-10-20 00:01:33][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,573 | MFU: 1.8% [2026-10-20 00:01:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.5752 [2026-10-20 00:01:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.0040 [2026-10-20 00:01:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.0613 [2026-10-20 00:01:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.5153 [2026-10-20 00:01:37][INFO][training.pretrain_engine] [Ph2|Step 134] loss=8.2071 | ppl=3667.05 | aux=0.02519 | lr=1.10e-04 | gnorm=1.391 | tok/s=29,283 | MFU=7.0% | step=143233ms | tokens=0.148B | mem=4.2/32.7GB | cost=$70.36/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 20:54 (140.9h) | Phase3: SKIPPED | SFT: Oct 26 01:39 (145.6h) | DPO (Final): Oct 26 02:13 (146.2h) [2026-10-20 00:01:37][INFO][training.pretrain_engine] Starting global step 135 (accumulating 32 micro-batches)... [2026-10-20 00:01:37][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 00:01:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.0977 [2026-10-20 00:01:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.0972 [2026-10-20 00:01:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.0640 [2026-10-20 00:01:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.1697 [2026-10-20 00:02:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.4260 [2026-10-20 00:02:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.5931 [2026-10-20 00:02:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.4026 [2026-10-20 00:02:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.9778 [2026-10-20 00:02:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.1231 [2026-10-20 00:02:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.2452 [2026-10-20 00:02:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.9802 [2026-10-20 00:02:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.0604 [2026-10-20 00:02:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.7631 [2026-10-20 00:02:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.3042 [2026-10-20 00:02:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.1193 [2026-10-20 00:02:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.2537 [2026-10-20 00:03:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.1130 [2026-10-20 00:03:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.0897 [2026-10-20 00:03:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.3109 [2026-10-20 00:03:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.1266 [2026-10-20 00:03:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.7873 [2026-10-20 00:03:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.5297 [2026-10-20 00:03:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.8455 [2026-10-20 00:03:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.7378 [2026-10-20 00:03:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.1222 [2026-10-20 00:03:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.0378 [2026-10-20 00:03:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.4022 [2026-10-20 00:03:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.3411 [2026-10-20 00:03:56][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,571 | MFU: 1.8% [2026-10-20 00:04:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.3370 [2026-10-20 00:04:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.0279 [2026-10-20 00:04:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.2769 [2026-10-20 00:04:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.2273 [2026-10-20 00:04:00][INFO][training.pretrain_engine] [Ph2|Step 135] loss=8.3206 | ppl=4107.56 | aux=0.02524 | lr=1.10e-04 | gnorm=0.781 | tok/s=29,284 | MFU=7.0% | step=143227ms | tokens=0.149B | mem=4.2/32.7GB | cost=$70.88/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 20:56 (140.9h) | Phase3: SKIPPED | SFT: Oct 26 01:40 (145.6h) | DPO (Final): Oct 26 02:14 (146.2h) [2026-10-20 00:04:04][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step135 | Tokens: 0.149B | Spent: $70.90 | Remaining: $329.10 [2026-10-20 00:04:04][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step135 [2026-10-20 00:04:04][INFO][training.pretrain_engine] Starting global step 136 (accumulating 32 micro-batches)... [2026-10-20 00:04:04][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 00:04:04][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 00:04:04][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 00:04:04][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 00:04:04][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step135 to Phase-Technologies/forge-500m-pretrain (path: phase2_step135)... [2026-10-20 00:04:04][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 00:04:13][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 00:04:13][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step135 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 00:04:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.3711 [2026-10-20 00:04:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.1131 [2026-10-20 00:04:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.9663 [2026-10-20 00:04:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.0466 [2026-10-20 00:04:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.4940 [2026-10-20 00:04:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.6507 [2026-10-20 00:04:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.8752 [2026-10-20 00:04:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.2396 [2026-10-20 00:04:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.9939 [2026-10-20 00:04:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.8080 [2026-10-20 00:04:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.0749 [2026-10-20 00:04:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.0490 [2026-10-20 00:05:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.5116 [2026-10-20 00:05:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.2123 [2026-10-20 00:05:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.0467 [2026-10-20 00:05:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.2336 [2026-10-20 00:05:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.5171 [2026-10-20 00:05:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.4861 [2026-10-20 00:05:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.6195 [2026-10-20 00:05:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.5451 [2026-10-20 00:05:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.3527 [2026-10-20 00:05:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.2185 [2026-10-20 00:05:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.9173 [2026-10-20 00:05:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.0965 [2026-10-20 00:06:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.9224 [2026-10-20 00:06:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.8610 [2026-10-20 00:06:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.3602 [2026-10-20 00:06:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.6725 [2026-10-20 00:06:22][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,560 | MFU: 1.8% [2026-10-20 00:06:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.9328 [2026-10-20 00:06:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.6899 [2026-10-20 00:06:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.7889 [2026-10-20 00:06:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.6454 [2026-10-20 00:06:27][INFO][training.pretrain_engine] [Ph2|Step 136] loss=8.1939 | ppl=3618.65 | aux=0.02522 | lr=1.11e-04 | gnorm=0.852 | tok/s=29,285 | MFU=7.0% | step=143223ms | tokens=0.150B | mem=4.2/32.7GB | cost=$71.42/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 20:57 (140.9h) | Phase3: SKIPPED | SFT: Oct 26 01:42 (145.6h) | DPO (Final): Oct 26 02:16 (146.2h) [2026-10-20 00:06:27][INFO][training.pretrain_engine] Starting global step 137 (accumulating 32 micro-batches)... [2026-10-20 00:06:27][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 00:06:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.1652 [2026-10-20 00:06:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.2867 [2026-10-20 00:06:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.3360 [2026-10-20 00:06:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.2232 [2026-10-20 00:07:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.3310 [2026-10-20 00:07:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.0466 [2026-10-20 00:07:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.4710 [2026-10-20 00:07:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.0455 [2026-10-20 00:07:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.3351 [2026-10-20 00:07:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.7657 [2026-10-20 00:07:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.2799 [2026-10-20 00:07:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.0968 [2026-10-20 00:07:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.1246 [2026-10-20 00:07:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.9969 [2026-10-20 00:07:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.9774 [2026-10-20 00:07:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.9933 [2026-10-20 00:07:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.3269 [2026-10-20 00:07:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.3766 [2026-10-20 00:07:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.6641 [2026-10-20 00:07:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.4406 [2026-10-20 00:08:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.0766 [2026-10-20 00:08:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.2787 [2026-10-20 00:08:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.1617 [2026-10-20 00:08:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.1363 [2026-10-20 00:08:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.5003 [2026-10-20 00:08:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.7670 [2026-10-20 00:08:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.7426 [2026-10-20 00:08:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.0338 [2026-10-20 00:08:45][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,582 | MFU: 1.8% [2026-10-20 00:08:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.1164 [2026-10-20 00:08:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.0078 [2026-10-20 00:08:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.8955 [2026-10-20 00:08:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.0076 [2026-10-20 00:08:50][INFO][training.pretrain_engine] [Ph2|Step 137] loss=8.2425 | ppl=3799.17 | aux=0.02518 | lr=1.12e-04 | gnorm=0.996 | tok/s=29,285 | MFU=7.0% | step=143222ms | tokens=0.151B | mem=4.2/32.7GB | cost=$71.94/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 20:59 (140.8h) | Phase3: SKIPPED | SFT: Oct 26 01:44 (145.6h) | DPO (Final): Oct 26 02:18 (146.2h) [2026-10-20 00:08:50][INFO][training.pretrain_engine] Starting global step 138 (accumulating 32 micro-batches)... [2026-10-20 00:08:50][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 00:09:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.8886 [2026-10-20 00:09:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.9745 [2026-10-20 00:09:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.6900 [2026-10-20 00:09:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.7585 [2026-10-20 00:09:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.8268 [2026-10-20 00:09:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.5017 [2026-10-20 00:09:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.1075 [2026-10-20 00:09:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.8490 [2026-10-20 00:09:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.4588 [2026-10-20 00:09:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.3923 [2026-10-20 00:09:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.4368 [2026-10-20 00:09:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.4246 [2026-10-20 00:10:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.9183 [2026-10-20 00:10:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.1923 [2026-10-20 00:10:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.0064 [2026-10-20 00:10:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.0099 [2026-10-20 00:10:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.2625 [2026-10-20 00:10:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.8795 [2026-10-20 00:10:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.6228 [2026-10-20 00:10:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.0515 [2026-10-20 00:10:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.0282 [2026-10-20 00:10:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.7560 [2026-10-20 00:10:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.0400 [2026-10-20 00:10:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.3637 [2026-10-20 00:10:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.0513 [2026-10-20 00:10:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.2945 [2026-10-20 00:10:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.4973 [2026-10-20 00:10:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.7285 [2026-10-20 00:11:09][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,574 | MFU: 1.8% [2026-10-20 00:11:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.7920 [2026-10-20 00:11:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.2609 [2026-10-20 00:11:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.1324 [2026-10-20 00:11:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.0319 [2026-10-20 00:11:13][INFO][training.pretrain_engine] [Ph2|Step 138] loss=8.1760 | ppl=3554.54 | aux=0.02523 | lr=1.12e-04 | gnorm=0.984 | tok/s=29,286 | MFU=7.0% | step=143217ms | tokens=0.152B | mem=4.2/32.7GB | cost=$72.47/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 21:01 (140.8h) | Phase3: SKIPPED | SFT: Oct 26 01:45 (145.6h) | DPO (Final): Oct 26 02:19 (146.1h) [2026-10-20 00:11:13][INFO][training.pretrain_engine] Starting global step 139 (accumulating 32 micro-batches)... [2026-10-20 00:11:13][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 00:11:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.9106 [2026-10-20 00:11:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.1223 [2026-10-20 00:11:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.1483 [2026-10-20 00:11:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.3764 [2026-10-20 00:11:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.6957 [2026-10-20 00:11:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.5369 [2026-10-20 00:11:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.8545 [2026-10-20 00:11:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.1246 [2026-10-20 00:12:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.1018 [2026-10-20 00:12:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.9753 [2026-10-20 00:12:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.0101 [2026-10-20 00:12:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.5562 [2026-10-20 00:12:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.0561 [2026-10-20 00:12:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.8486 [2026-10-20 00:12:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.1780 [2026-10-20 00:12:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.3310 [2026-10-20 00:12:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.8895 [2026-10-20 00:12:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.9648 [2026-10-20 00:12:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.9556 [2026-10-20 00:12:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.3431 [2026-10-20 00:13:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.1761 [2026-10-20 00:13:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.2135 [2026-10-20 00:13:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.1696 [2026-10-20 00:13:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.1626 [2026-10-20 00:13:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.3367 [2026-10-20 00:13:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.7497 [2026-10-20 00:13:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.6653 [2026-10-20 00:13:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=9.0408 [2026-10-20 00:13:32][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,573 | MFU: 1.8% [2026-10-20 00:13:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.3244 [2026-10-20 00:13:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.2527 [2026-10-20 00:13:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.4606 [2026-10-20 00:13:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.5663 [2026-10-20 00:13:36][INFO][training.pretrain_engine] [Ph2|Step 139] loss=8.3276 | ppl=4136.60 | aux=0.02525 | lr=1.13e-04 | gnorm=0.863 | tok/s=29,287 | MFU=7.0% | step=143214ms | tokens=0.153B | mem=4.2/32.7GB | cost=$72.99/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 21:02 (140.8h) | Phase3: SKIPPED | SFT: Oct 26 01:47 (145.6h) | DPO (Final): Oct 26 02:21 (146.1h) [2026-10-20 00:13:36][INFO][training.pretrain_engine] Starting global step 140 (accumulating 32 micro-batches)... [2026-10-20 00:13:36][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 00:13:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.0281 [2026-10-20 00:13:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.9818 [2026-10-20 00:13:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.8447 [2026-10-20 00:13:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.1490 [2026-10-20 00:14:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.0766 [2026-10-20 00:14:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.2344 [2026-10-20 00:14:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.4009 [2026-10-20 00:14:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.0025 [2026-10-20 00:14:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.9886 [2026-10-20 00:14:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.6386 [2026-10-20 00:14:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.0196 [2026-10-20 00:14:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.9483 [2026-10-20 00:14:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.0597 [2026-10-20 00:14:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.1030 [2026-10-20 00:14:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.2099 [2026-10-20 00:14:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.1877 [2026-10-20 00:15:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.2236 [2026-10-20 00:15:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.1979 [2026-10-20 00:15:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.1380 [2026-10-20 00:15:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=9.1001 [2026-10-20 00:15:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.9723 [2026-10-20 00:15:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.9053 [2026-10-20 00:15:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.3716 [2026-10-20 00:15:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.8443 [2026-10-20 00:15:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.9746 [2026-10-20 00:15:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.1423 [2026-10-20 00:15:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.9224 [2026-10-20 00:15:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.0335 [2026-10-20 00:15:56][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,530 | MFU: 1.8% [2026-10-20 00:16:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.4459 [2026-10-20 00:16:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.2305 [2026-10-20 00:16:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.0501 [2026-10-20 00:16:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.2043 [2026-10-20 00:16:00][INFO][training.pretrain_engine] [Ph2|Step 140] loss=8.1512 | ppl=3467.49 | aux=0.02520 | lr=1.13e-04 | gnorm=0.695 | tok/s=29,285 | MFU=7.0% | step=143223ms | tokens=0.154B | mem=4.2/32.7GB | cost=$73.51/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 21:05 (140.8h) | Phase3: SKIPPED | SFT: Oct 26 01:49 (145.6h) | DPO (Final): Oct 26 02:23 (146.1h) [2026-10-20 00:16:00][INFO][training.pretrain_engine] Starting global step 141 (accumulating 32 micro-batches)... [2026-10-20 00:16:00][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 00:16:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.6640 [2026-10-20 00:16:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.6959 [2026-10-20 00:16:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.4813 [2026-10-20 00:16:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.9774 [2026-10-20 00:16:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.9869 [2026-10-20 00:16:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.1832 [2026-10-20 00:16:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.9623 [2026-10-20 00:16:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.2863 [2026-10-20 00:16:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.4266 [2026-10-20 00:16:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.1346 [2026-10-20 00:16:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.3062 [2026-10-20 00:16:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.1977 [2026-10-20 00:17:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.9722 [2026-10-20 00:17:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.1392 [2026-10-20 00:17:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.8445 [2026-10-20 00:17:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.0842 [2026-10-20 00:17:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.9881 [2026-10-20 00:17:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.8934 [2026-10-20 00:17:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.8510 [2026-10-20 00:17:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.0118 [2026-10-20 00:17:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.5094 [2026-10-20 00:17:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.0820 [2026-10-20 00:17:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.1716 [2026-10-20 00:17:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.2159 [2026-10-20 00:18:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.2477 [2026-10-20 00:18:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.9427 [2026-10-20 00:18:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.9821 [2026-10-20 00:18:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.8906 [2026-10-20 00:18:19][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,552 | MFU: 1.8% [2026-10-20 00:18:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.7962 [2026-10-20 00:18:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.4132 [2026-10-20 00:18:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.4782 [2026-10-20 00:18:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.4904 [2026-10-20 00:18:24][INFO][training.pretrain_engine] [Ph2|Step 141] loss=8.1234 | ppl=3372.52 | aux=0.02519 | lr=1.14e-04 | gnorm=0.977 | tok/s=29,284 | MFU=7.0% | step=143229ms | tokens=0.155B | mem=4.2/32.7GB | cost=$74.04/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 21:07 (140.8h) | Phase3: SKIPPED | SFT: Oct 26 01:51 (145.6h) | DPO (Final): Oct 26 02:25 (146.1h) [2026-10-20 00:18:24][INFO][training.pretrain_engine] Starting global step 142 (accumulating 32 micro-batches)... [2026-10-20 00:18:24][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 00:18:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.4104 [2026-10-20 00:18:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.3857 [2026-10-20 00:18:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.7383 [2026-10-20 00:18:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.2541 [2026-10-20 00:18:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.0575 [2026-10-20 00:18:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.9157 [2026-10-20 00:18:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.1792 [2026-10-20 00:19:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.2005 [2026-10-20 00:19:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.7824 [2026-10-20 00:19:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.8567 [2026-10-20 00:19:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.3021 [2026-10-20 00:19:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.9722 [2026-10-20 00:19:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.2399 [2026-10-20 00:19:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.1335 [2026-10-20 00:19:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.4472 [2026-10-20 00:19:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.9890 [2026-10-20 00:19:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.9694 [2026-10-20 00:19:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.6105 [2026-10-20 00:19:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.9728 [2026-10-20 00:19:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.3780 [2026-10-20 00:20:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.0355 [2026-10-20 00:20:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.0304 [2026-10-20 00:20:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.1796 [2026-10-20 00:20:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.9147 [2026-10-20 00:20:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.9174 [2026-10-20 00:20:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.0625 [2026-10-20 00:20:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.0009 [2026-10-20 00:20:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.0547 [2026-10-20 00:20:43][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,520 | MFU: 1.8% [2026-10-20 00:20:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.9891 [2026-10-20 00:20:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.7502 [2026-10-20 00:20:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.0724 [2026-10-20 00:20:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.8359 [2026-10-20 00:20:48][INFO][training.pretrain_engine] [Ph2|Step 142] loss=8.1153 | ppl=3345.31 | aux=0.02519 | lr=1.14e-04 | gnorm=1.078 | tok/s=29,282 | MFU=7.0% | step=143239ms | tokens=0.156B | mem=4.2/32.7GB | cost=$74.57/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 21:09 (140.8h) | Phase3: SKIPPED | SFT: Oct 26 01:54 (145.6h) | DPO (Final): Oct 26 02:28 (146.1h) [2026-10-20 00:20:48][INFO][training.pretrain_engine] Starting global step 143 (accumulating 32 micro-batches)... [2026-10-20 00:20:48][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 00:21:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.1210 [2026-10-20 00:21:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.3055 [2026-10-20 00:21:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.4783 [2026-10-20 00:21:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.0004 [2026-10-20 00:21:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.3034 [2026-10-20 00:21:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.0543 [2026-10-20 00:21:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.1506 [2026-10-20 00:21:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.1182 [2026-10-20 00:21:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.8977 [2026-10-20 00:21:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.1528 [2026-10-20 00:21:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.8775 [2026-10-20 00:21:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.1109 [2026-10-20 00:21:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.8985 [2026-10-20 00:21:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.3262 [2026-10-20 00:21:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.8978 [2026-10-20 00:21:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.4607 [2026-10-20 00:22:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.9568 [2026-10-20 00:22:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.8634 [2026-10-20 00:22:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.8481 [2026-10-20 00:22:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.8613 [2026-10-20 00:22:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.7191 [2026-10-20 00:22:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.0567 [2026-10-20 00:22:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.8764 [2026-10-20 00:22:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.1985 [2026-10-20 00:22:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.9475 [2026-10-20 00:22:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.0010 [2026-10-20 00:22:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.9417 [2026-10-20 00:22:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.0328 [2026-10-20 00:23:06][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,555 | MFU: 1.8% [2026-10-20 00:23:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.0101 [2026-10-20 00:23:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.1365 [2026-10-20 00:23:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.0352 [2026-10-20 00:23:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.4430 [2026-10-20 00:23:11][INFO][training.pretrain_engine] [Ph2|Step 143] loss=8.1527 | ppl=3472.91 | aux=0.02520 | lr=1.15e-04 | gnorm=0.871 | tok/s=29,280 | MFU=7.0% | step=143246ms | tokens=0.157B | mem=4.2/32.7GB | cost=$75.09/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 21:11 (140.8h) | Phase3: SKIPPED | SFT: Oct 26 01:56 (145.6h) | DPO (Final): Oct 26 02:30 (146.1h) [2026-10-20 00:23:11][INFO][training.pretrain_engine] Starting global step 144 (accumulating 32 micro-batches)... [2026-10-20 00:23:11][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 00:23:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.9314 [2026-10-20 00:23:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.8651 [2026-10-20 00:23:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.1597 [2026-10-20 00:23:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.2281 [2026-10-20 00:23:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.3434 [2026-10-20 00:23:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.2621 [2026-10-20 00:23:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.7164 [2026-10-20 00:23:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.2611 [2026-10-20 00:24:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.7762 [2026-10-20 00:24:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.9651 [2026-10-20 00:24:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.9256 [2026-10-20 00:24:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.0175 [2026-10-20 00:24:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.2320 [2026-10-20 00:24:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.0751 [2026-10-20 00:24:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.1614 [2026-10-20 00:24:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.5021 [2026-10-20 00:24:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.9952 [2026-10-20 00:24:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.8750 [2026-10-20 00:24:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.4683 [2026-10-20 00:24:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.1984 [2026-10-20 00:24:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.8113 [2026-10-20 00:24:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.0529 [2026-10-20 00:24:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.0311 [2026-10-20 00:24:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.1035 [2026-10-20 00:25:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.2982 [2026-10-20 00:25:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.3936 [2026-10-20 00:25:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.4656 [2026-10-20 00:25:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.7845 [2026-10-20 00:25:30][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,564 | MFU: 1.8% [2026-10-20 00:25:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.9161 [2026-10-20 00:25:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.2789 [2026-10-20 00:25:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.8391 [2026-10-20 00:25:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.0286 [2026-10-20 00:25:34][INFO][training.pretrain_engine] [Ph2|Step 144] loss=8.3099 | ppl=4064.04 | aux=0.02525 | lr=1.16e-04 | gnorm=0.832 | tok/s=29,279 | MFU=7.0% | step=143251ms | tokens=0.158B | mem=4.2/32.7GB | cost=$75.61/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 21:13 (140.8h) | Phase3: SKIPPED | SFT: Oct 26 01:58 (145.5h) | DPO (Final): Oct 26 02:32 (146.1h) [2026-10-20 00:25:37][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step144 | Tokens: 0.158B | Spent: $75.63 | Remaining: $324.37 [2026-10-20 00:25:37][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step144 [2026-10-20 00:25:37][INFO][training.pretrain_engine] Starting global step 145 (accumulating 32 micro-batches)... [2026-10-20 00:25:37][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 00:25:37][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 00:25:37][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 00:25:38][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 00:25:38][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step144 to Phase-Technologies/forge-500m-pretrain (path: phase2_step144)... [2026-10-20 00:25:38][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 00:25:47][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 00:25:47][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step144 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 00:25:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.0481 [2026-10-20 00:25:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.8882 [2026-10-20 00:25:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.9637 [2026-10-20 00:25:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.9195 [2026-10-20 00:26:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.8399 [2026-10-20 00:26:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.7896 [2026-10-20 00:26:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.8392 [2026-10-20 00:26:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.7240 [2026-10-20 00:26:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.2149 [2026-10-20 00:26:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.4010 [2026-10-20 00:26:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.3006 [2026-10-20 00:26:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=9.2922 [2026-10-20 00:26:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.8978 [2026-10-20 00:26:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.7323 [2026-10-20 00:26:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.3371 [2026-10-20 00:26:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.7781 [2026-10-20 00:27:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.3586 [2026-10-20 00:27:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.8942 [2026-10-20 00:27:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.7243 [2026-10-20 00:27:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.0401 [2026-10-20 00:27:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.3217 [2026-10-20 00:27:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.2934 [2026-10-20 00:27:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.6687 [2026-10-20 00:27:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.3546 [2026-10-20 00:27:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.9089 [2026-10-20 00:27:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.9842 [2026-10-20 00:27:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.8869 [2026-10-20 00:27:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.6786 [2026-10-20 00:27:56][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,558 | MFU: 1.8% [2026-10-20 00:28:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.9965 [2026-10-20 00:28:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.7670 [2026-10-20 00:28:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.9125 [2026-10-20 00:28:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.9452 [2026-10-20 00:28:01][INFO][training.pretrain_engine] [Ph2|Step 145] loss=8.0853 | ppl=3246.45 | aux=0.02519 | lr=1.16e-04 | gnorm=0.969 | tok/s=29,278 | MFU=7.0% | step=143257ms | tokens=0.159B | mem=4.2/32.7GB | cost=$76.15/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 21:16 (140.8h) | Phase3: SKIPPED | SFT: Oct 26 02:00 (145.5h) | DPO (Final): Oct 26 02:34 (146.1h) [2026-10-20 00:28:01][INFO][training.pretrain_engine] Starting global step 146 (accumulating 32 micro-batches)... [2026-10-20 00:28:01][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 00:28:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.4472 [2026-10-20 00:28:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.5510 [2026-10-20 00:28:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.7142 [2026-10-20 00:28:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.9263 [2026-10-20 00:28:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.2852 [2026-10-20 00:28:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.2761 [2026-10-20 00:28:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.1072 [2026-10-20 00:28:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.1388 [2026-10-20 00:28:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.9636 [2026-10-20 00:28:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.9487 [2026-10-20 00:28:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.8409 [2026-10-20 00:28:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.8063 [2026-10-20 00:29:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.4869 [2026-10-20 00:29:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.5792 [2026-10-20 00:29:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.7361 [2026-10-20 00:29:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.6695 [2026-10-20 00:29:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.4132 [2026-10-20 00:29:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.6514 [2026-10-20 00:29:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.6502 [2026-10-20 00:29:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.7503 [2026-10-20 00:29:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.8853 [2026-10-20 00:29:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.1954 [2026-10-20 00:29:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.8108 [2026-10-20 00:29:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.0272 [2026-10-20 00:30:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.2626 [2026-10-20 00:30:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.0033 [2026-10-20 00:30:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.2083 [2026-10-20 00:30:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.2019 [2026-10-20 00:30:19][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,583 | MFU: 1.8% [2026-10-20 00:30:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.9732 [2026-10-20 00:30:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.6613 [2026-10-20 00:30:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.7874 [2026-10-20 00:30:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.7996 [2026-10-20 00:30:24][INFO][training.pretrain_engine] [Ph2|Step 146] loss=8.1063 | ppl=3315.25 | aux=0.02518 | lr=1.17e-04 | gnorm=0.848 | tok/s=29,278 | MFU=7.0% | step=143256ms | tokens=0.160B | mem=4.2/32.7GB | cost=$76.67/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 21:17 (140.8h) | Phase3: SKIPPED | SFT: Oct 26 02:02 (145.5h) | DPO (Final): Oct 26 02:36 (146.1h) [2026-10-20 00:30:24][INFO][training.pretrain_engine] Starting global step 147 (accumulating 32 micro-batches)... [2026-10-20 00:30:24][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 00:30:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.3436 [2026-10-20 00:30:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.1189 [2026-10-20 00:30:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.0246 [2026-10-20 00:30:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.8820 [2026-10-20 00:30:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.1252 [2026-10-20 00:30:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.9507 [2026-10-20 00:30:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.9225 [2026-10-20 00:30:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.6385 [2026-10-20 00:31:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.3063 [2026-10-20 00:31:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.9810 [2026-10-20 00:31:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.0230 [2026-10-20 00:31:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.9188 [2026-10-20 00:31:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.9001 [2026-10-20 00:31:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.8263 [2026-10-20 00:31:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.6736 [2026-10-20 00:31:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.6251 [2026-10-20 00:31:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.1613 [2026-10-20 00:31:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.9294 [2026-10-20 00:31:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.1510 [2026-10-20 00:31:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.2296 [2026-10-20 00:32:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.8940 [2026-10-20 00:32:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.7957 [2026-10-20 00:32:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.9742 [2026-10-20 00:32:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.8437 [2026-10-20 00:32:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.9542 [2026-10-20 00:32:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.1671 [2026-10-20 00:32:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.8071 [2026-10-20 00:32:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.8522 [2026-10-20 00:32:42][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,599 | MFU: 1.8% [2026-10-20 00:32:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.9853 [2026-10-20 00:32:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.9555 [2026-10-20 00:32:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.7931 [2026-10-20 00:32:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.7174 [2026-10-20 00:32:47][INFO][training.pretrain_engine] [Ph2|Step 147] loss=8.0532 | ppl=3143.81 | aux=0.02517 | lr=1.17e-04 | gnorm=0.715 | tok/s=29,277 | MFU=7.0% | step=143261ms | tokens=0.161B | mem=4.2/32.7GB | cost=$77.19/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 21:19 (140.8h) | Phase3: SKIPPED | SFT: Oct 26 02:04 (145.5h) | DPO (Final): Oct 26 02:38 (146.1h) [2026-10-20 00:32:47][INFO][training.pretrain_engine] Starting global step 148 (accumulating 32 micro-batches)... [2026-10-20 00:32:47][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 00:33:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.7251 [2026-10-20 00:33:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.0228 [2026-10-20 00:33:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.8307 [2026-10-20 00:33:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.9658 [2026-10-20 00:33:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.0620 [2026-10-20 00:33:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.9924 [2026-10-20 00:33:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.8394 [2026-10-20 00:33:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.8981 [2026-10-20 00:33:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.5328 [2026-10-20 00:33:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.1382 [2026-10-20 00:33:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.6398 [2026-10-20 00:33:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.7737 [2026-10-20 00:33:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.1165 [2026-10-20 00:33:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.6248 [2026-10-20 00:33:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.5926 [2026-10-20 00:33:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.8397 [2026-10-20 00:34:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.2891 [2026-10-20 00:34:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.9424 [2026-10-20 00:34:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.0050 [2026-10-20 00:34:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.0844 [2026-10-20 00:34:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.0083 [2026-10-20 00:34:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.2904 [2026-10-20 00:34:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.9110 [2026-10-20 00:34:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.6454 [2026-10-20 00:34:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.7811 [2026-10-20 00:34:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.0015 [2026-10-20 00:34:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.7165 [2026-10-20 00:34:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.8020 [2026-10-20 00:35:05][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,599 | MFU: 1.8% [2026-10-20 00:35:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.0641 [2026-10-20 00:35:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.1360 [2026-10-20 00:35:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.0945 [2026-10-20 00:35:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.7781 [2026-10-20 00:35:10][INFO][training.pretrain_engine] [Ph2|Step 148] loss=8.0724 | ppl=3204.63 | aux=0.02517 | lr=1.18e-04 | gnorm=0.910 | tok/s=29,276 | MFU=7.0% | step=143266ms | tokens=0.163B | mem=4.2/32.7GB | cost=$77.72/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 21:21 (140.8h) | Phase3: SKIPPED | SFT: Oct 26 02:06 (145.5h) | DPO (Final): Oct 26 02:40 (146.1h) [2026-10-20 00:35:10][INFO][training.pretrain_engine] Starting global step 149 (accumulating 32 micro-batches)... [2026-10-20 00:35:10][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 00:35:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.7753 [2026-10-20 00:35:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.7223 [2026-10-20 00:35:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.7008 [2026-10-20 00:35:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.6592 [2026-10-20 00:35:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.0207 [2026-10-20 00:35:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.6951 [2026-10-20 00:35:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.0609 [2026-10-20 00:35:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.9124 [2026-10-20 00:36:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.4797 [2026-10-20 00:36:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.7192 [2026-10-20 00:36:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.5851 [2026-10-20 00:36:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.8384 [2026-10-20 00:36:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.2756 [2026-10-20 00:36:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.3207 [2026-10-20 00:36:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.4154 [2026-10-20 00:36:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.1215 [2026-10-20 00:36:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.9928 [2026-10-20 00:36:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.5626 [2026-10-20 00:36:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.0275 [2026-10-20 00:36:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.9287 [2026-10-20 00:36:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.8106 [2026-10-20 00:36:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.1652 [2026-10-20 00:36:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1453 [2026-10-20 00:36:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.8968 [2026-10-20 00:37:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.1073 [2026-10-20 00:37:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.2420 [2026-10-20 00:37:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.2085 [2026-10-20 00:37:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.8141 [2026-10-20 00:37:29][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,553 | MFU: 1.8% [2026-10-20 00:37:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.5529 [2026-10-20 00:37:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.8377 [2026-10-20 00:37:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.4000 [2026-10-20 00:37:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.2007 [2026-10-20 00:37:33][INFO][training.pretrain_engine] [Ph2|Step 149] loss=7.9370 | ppl=2798.91 | aux=0.02521 | lr=1.18e-04 | gnorm=1.070 | tok/s=29,276 | MFU=7.0% | step=143270ms | tokens=0.164B | mem=4.2/32.7GB | cost=$78.24/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 21:23 (140.8h) | Phase3: SKIPPED | SFT: Oct 26 02:08 (145.5h) | DPO (Final): Oct 26 02:42 (146.1h) [2026-10-20 00:37:33][INFO][training.pretrain_engine] Starting global step 150 (accumulating 32 micro-batches)... [2026-10-20 00:37:33][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 00:37:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.4419 [2026-10-20 00:37:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.4480 [2026-10-20 00:37:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.5340 [2026-10-20 00:37:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.6549 [2026-10-20 00:38:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.6733 [2026-10-20 00:38:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.6333 [2026-10-20 00:38:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.9278 [2026-10-20 00:38:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.6909 [2026-10-20 00:38:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.8815 [2026-10-20 00:38:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.9768 [2026-10-20 00:38:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.1492 [2026-10-20 00:38:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.0630 [2026-10-20 00:38:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=9.7799 [2026-10-20 00:38:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.0779 [2026-10-20 00:38:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.5617 [2026-10-20 00:38:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.6750 [2026-10-20 00:39:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.7630 [2026-10-20 00:39:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.7403 [2026-10-20 00:39:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9504 [2026-10-20 00:39:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.8563 [2026-10-20 00:39:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.1856 [2026-10-20 00:39:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.4381 [2026-10-20 00:39:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.3484 [2026-10-20 00:39:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.3548 [2026-10-20 00:39:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.1842 [2026-10-20 00:39:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.2443 [2026-10-20 00:39:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.3829 [2026-10-20 00:39:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.5042 [2026-10-20 00:39:52][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,581 | MFU: 1.8% [2026-10-20 00:39:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.8308 [2026-10-20 00:39:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.7804 [2026-10-20 00:39:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.9978 [2026-10-20 00:39:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.7458 [2026-10-20 00:39:57][INFO][training.pretrain_engine] [Ph2|Step 150] loss=8.0408 | ppl=3104.99 | aux=0.02520 | lr=1.19e-04 | gnorm=1.023 | tok/s=29,274 | MFU=7.0% | step=143277ms | tokens=0.165B | mem=4.2/32.7GB | cost=$78.77/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 21:26 (140.8h) | Phase3: SKIPPED | SFT: Oct 26 02:10 (145.5h) | DPO (Final): Oct 26 02:45 (146.1h) [2026-10-20 00:39:57][INFO][training.pretrain_engine] Starting global step 151 (accumulating 32 micro-batches)... [2026-10-20 00:39:57][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 00:40:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.3741 [2026-10-20 00:40:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.9653 [2026-10-20 00:40:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.0896 [2026-10-20 00:40:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.1230 [2026-10-20 00:40:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.1090 [2026-10-20 00:40:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.7130 [2026-10-20 00:40:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.0072 [2026-10-20 00:40:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.8552 [2026-10-20 00:40:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.6437 [2026-10-20 00:40:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4316 [2026-10-20 00:40:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.8038 [2026-10-20 00:40:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4652 [2026-10-20 00:41:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.0057 [2026-10-20 00:41:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.8772 [2026-10-20 00:41:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.7622 [2026-10-20 00:41:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.8393 [2026-10-20 00:41:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.2879 [2026-10-20 00:41:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.5321 [2026-10-20 00:41:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.2894 [2026-10-20 00:41:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.8785 [2026-10-20 00:41:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.7675 [2026-10-20 00:41:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.2545 [2026-10-20 00:41:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.9404 [2026-10-20 00:41:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.8578 [2026-10-20 00:42:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.9676 [2026-10-20 00:42:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.9069 [2026-10-20 00:42:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.9454 [2026-10-20 00:42:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.8034 [2026-10-20 00:42:16][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,545 | MFU: 1.8% [2026-10-20 00:42:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.7439 [2026-10-20 00:42:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.0454 [2026-10-20 00:42:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.9584 [2026-10-20 00:42:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.9348 [2026-10-20 00:42:20][INFO][training.pretrain_engine] [Ph2|Step 151] loss=7.9094 | ppl=2722.64 | aux=0.02520 | lr=1.19e-04 | gnorm=1.008 | tok/s=29,273 | MFU=7.0% | step=143283ms | tokens=0.166B | mem=4.2/32.7GB | cost=$79.29/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 21:28 (140.8h) | Phase3: SKIPPED | SFT: Oct 26 02:13 (145.5h) | DPO (Final): Oct 26 02:47 (146.1h) [2026-10-20 00:42:20][INFO][training.pretrain_engine] Starting global step 152 (accumulating 32 micro-batches)... [2026-10-20 00:42:20][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 00:42:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.8253 [2026-10-20 00:42:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.0546 [2026-10-20 00:42:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.7556 [2026-10-20 00:42:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.0712 [2026-10-20 00:42:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.8684 [2026-10-20 00:42:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.5582 [2026-10-20 00:42:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.7817 [2026-10-20 00:42:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.6232 [2026-10-20 00:43:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.7013 [2026-10-20 00:43:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.7793 [2026-10-20 00:43:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.7400 [2026-10-20 00:43:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.0342 [2026-10-20 00:43:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.3820 [2026-10-20 00:43:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.5390 [2026-10-20 00:43:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.9260 [2026-10-20 00:43:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.0878 [2026-10-20 00:43:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.5414 [2026-10-20 00:43:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.8361 [2026-10-20 00:43:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3886 [2026-10-20 00:43:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.4040 [2026-10-20 00:44:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.7079 [2026-10-20 00:44:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.7074 [2026-10-20 00:44:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.3813 [2026-10-20 00:44:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.4876 [2026-10-20 00:44:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.7158 [2026-10-20 00:44:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.7946 [2026-10-20 00:44:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.9259 [2026-10-20 00:44:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.6290 [2026-10-20 00:44:39][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,569 | MFU: 1.8% [2026-10-20 00:44:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.7032 [2026-10-20 00:44:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.0877 [2026-10-20 00:44:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.0183 [2026-10-20 00:44:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.6496 [2026-10-20 00:44:43][INFO][training.pretrain_engine] [Ph2|Step 152] loss=7.9968 | ppl=2971.57 | aux=0.02523 | lr=1.20e-04 | gnorm=0.945 | tok/s=29,272 | MFU=7.0% | step=143288ms | tokens=0.167B | mem=4.2/32.7GB | cost=$79.81/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 21:30 (140.8h) | Phase3: SKIPPED | SFT: Oct 26 02:15 (145.5h) | DPO (Final): Oct 26 02:49 (146.1h) [2026-10-20 00:44:43][INFO][training.pretrain_engine] Starting global step 153 (accumulating 32 micro-batches)... [2026-10-20 00:44:43][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 00:45:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.0372 [2026-10-20 00:45:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.7859 [2026-10-20 00:45:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.7377 [2026-10-20 00:45:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.9634 [2026-10-20 00:45:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.8602 [2026-10-20 00:45:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.8541 [2026-10-20 00:45:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.7976 [2026-10-20 00:45:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.6564 [2026-10-20 00:45:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.5312 [2026-10-20 00:45:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.6321 [2026-10-20 00:45:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.6162 [2026-10-20 00:45:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.6495 [2026-10-20 00:45:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.7953 [2026-10-20 00:45:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.8558 [2026-10-20 00:45:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.6263 [2026-10-20 00:45:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.7009 [2026-10-20 00:46:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.8030 [2026-10-20 00:46:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.7795 [2026-10-20 00:46:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.7088 [2026-10-20 00:46:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.6219 [2026-10-20 00:46:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.6229 [2026-10-20 00:46:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.5257 [2026-10-20 00:46:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.7778 [2026-10-20 00:46:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.5581 [2026-10-20 00:46:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.6532 [2026-10-20 00:46:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.1761 [2026-10-20 00:46:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.8699 [2026-10-20 00:46:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.2565 [2026-10-20 00:47:02][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,557 | MFU: 1.8% [2026-10-20 00:47:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.8744 [2026-10-20 00:47:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.1951 [2026-10-20 00:47:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.8519 [2026-10-20 00:47:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.1286 [2026-10-20 00:47:07][INFO][training.pretrain_engine] [Ph2|Step 153] loss=7.8229 | ppl=2497.21 | aux=0.02515 | lr=1.21e-04 | gnorm=0.648 | tok/s=29,272 | MFU=7.0% | step=143289ms | tokens=0.168B | mem=4.2/32.7GB | cost=$80.34/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 21:32 (140.8h) | Phase3: SKIPPED | SFT: Oct 26 02:16 (145.5h) | DPO (Final): Oct 26 02:51 (146.1h) [2026-10-20 00:47:10][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step153 | Tokens: 0.168B | Spent: $80.35 | Remaining: $319.65 [2026-10-20 00:47:10][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step153 [2026-10-20 00:47:10][INFO][training.pretrain_engine] Starting global step 154 (accumulating 32 micro-batches)... [2026-10-20 00:47:10][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 00:47:10][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 00:47:10][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 00:47:10][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 00:47:10][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step153 to Phase-Technologies/forge-500m-pretrain (path: phase2_step153)... [2026-10-20 00:47:10][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 00:47:19][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 00:47:19][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step153 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 00:47:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.1250 [2026-10-20 00:47:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.7452 [2026-10-20 00:47:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.2466 [2026-10-20 00:47:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.2648 [2026-10-20 00:47:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.6382 [2026-10-20 00:47:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.8222 [2026-10-20 00:47:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.7499 [2026-10-20 00:47:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.5584 [2026-10-20 00:48:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.7756 [2026-10-20 00:48:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.8652 [2026-10-20 00:48:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.5969 [2026-10-20 00:48:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.7311 [2026-10-20 00:48:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.8899 [2026-10-20 00:48:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.6261 [2026-10-20 00:48:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.7028 [2026-10-20 00:48:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.7473 [2026-10-20 00:48:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.1907 [2026-10-20 00:48:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.2291 [2026-10-20 00:48:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.8332 [2026-10-20 00:48:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.7860 [2026-10-20 00:48:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.6452 [2026-10-20 00:48:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.7046 [2026-10-20 00:48:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.3859 [2026-10-20 00:48:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.9212 [2026-10-20 00:49:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.7553 [2026-10-20 00:49:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.0391 [2026-10-20 00:49:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.8504 [2026-10-20 00:49:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.7668 [2026-10-20 00:49:29][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,543 | MFU: 1.8% [2026-10-20 00:49:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.6341 [2026-10-20 00:49:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.8485 [2026-10-20 00:49:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4613 [2026-10-20 00:49:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.5761 [2026-10-20 00:49:34][INFO][training.pretrain_engine] [Ph2|Step 154] loss=7.9659 | ppl=2881.12 | aux=0.02518 | lr=1.21e-04 | gnorm=0.977 | tok/s=29,271 | MFU=7.0% | step=143293ms | tokens=0.169B | mem=4.2/32.7GB | cost=$80.88/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 21:34 (140.7h) | Phase3: SKIPPED | SFT: Oct 26 02:19 (145.5h) | DPO (Final): Oct 26 02:53 (146.1h) [2026-10-20 00:49:34][INFO][training.pretrain_engine] Starting global step 155 (accumulating 32 micro-batches)... [2026-10-20 00:49:34][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 00:49:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.3221 [2026-10-20 00:49:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.1913 [2026-10-20 00:49:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.8655 [2026-10-20 00:49:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.5586 [2026-10-20 00:50:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.8429 [2026-10-20 00:50:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.9332 [2026-10-20 00:50:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.6334 [2026-10-20 00:50:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.9281 [2026-10-20 00:50:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.7872 [2026-10-20 00:50:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.7306 [2026-10-20 00:50:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.9115 [2026-10-20 00:50:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.0663 [2026-10-20 00:50:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.8616 [2026-10-20 00:50:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.8054 [2026-10-20 00:50:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.8617 [2026-10-20 00:50:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.9253 [2026-10-20 00:51:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.1823 [2026-10-20 00:51:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.8942 [2026-10-20 00:51:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.9977 [2026-10-20 00:51:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.9076 [2026-10-20 00:51:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.2451 [2026-10-20 00:51:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.2307 [2026-10-20 00:51:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.1298 [2026-10-20 00:51:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.1765 [2026-10-20 00:51:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.8036 [2026-10-20 00:51:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.8223 [2026-10-20 00:51:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.8073 [2026-10-20 00:51:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.4289 [2026-10-20 00:51:53][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,549 | MFU: 1.8% [2026-10-20 00:51:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.7913 [2026-10-20 00:51:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.6725 [2026-10-20 00:51:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.7520 [2026-10-20 00:51:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.7930 [2026-10-20 00:51:57][INFO][training.pretrain_engine] [Ph2|Step 155] loss=7.9923 | ppl=2958.23 | aux=0.02518 | lr=1.22e-04 | gnorm=0.914 | tok/s=29,271 | MFU=7.0% | step=143292ms | tokens=0.170B | mem=4.2/32.7GB | cost=$81.40/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 21:36 (140.7h) | Phase3: SKIPPED | SFT: Oct 26 02:20 (145.5h) | DPO (Final): Oct 26 02:54 (146.1h) [2026-10-20 00:51:57][INFO][training.pretrain_engine] Starting global step 156 (accumulating 32 micro-batches)... [2026-10-20 00:51:57][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 00:52:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.7511 [2026-10-20 00:52:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.7961 [2026-10-20 00:52:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=9.0936 [2026-10-20 00:52:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.7895 [2026-10-20 00:52:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.3672 [2026-10-20 00:52:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.2381 [2026-10-20 00:52:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.2848 [2026-10-20 00:52:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.2165 [2026-10-20 00:52:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.6822 [2026-10-20 00:52:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.8976 [2026-10-20 00:52:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.8267 [2026-10-20 00:52:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.5206 [2026-10-20 00:53:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.6262 [2026-10-20 00:53:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.5054 [2026-10-20 00:53:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.8876 [2026-10-20 00:53:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.6287 [2026-10-20 00:53:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.4914 [2026-10-20 00:53:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.7382 [2026-10-20 00:53:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.5026 [2026-10-20 00:53:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.6461 [2026-10-20 00:53:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.3988 [2026-10-20 00:53:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.7263 [2026-10-20 00:53:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.1193 [2026-10-20 00:53:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.5261 [2026-10-20 00:54:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.8313 [2026-10-20 00:54:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.7955 [2026-10-20 00:54:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.8647 [2026-10-20 00:54:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.9224 [2026-10-20 00:54:16][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,542 | MFU: 1.8% [2026-10-20 00:54:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4629 [2026-10-20 00:54:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.5643 [2026-10-20 00:54:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.8078 [2026-10-20 00:54:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.7232 [2026-10-20 00:54:21][INFO][training.pretrain_engine] [Ph2|Step 156] loss=7.8496 | ppl=2564.68 | aux=0.02516 | lr=1.22e-04 | gnorm=0.898 | tok/s=29,271 | MFU=7.0% | step=143294ms | tokens=0.171B | mem=4.2/32.7GB | cost=$81.93/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 21:38 (140.7h) | Phase3: SKIPPED | SFT: Oct 26 02:22 (145.5h) | DPO (Final): Oct 26 02:56 (146.0h) [2026-10-20 00:54:21][INFO][training.pretrain_engine] Starting global step 157 (accumulating 32 micro-batches)... [2026-10-20 00:54:21][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 00:54:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.6558 [2026-10-20 00:54:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.5328 [2026-10-20 00:54:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.7728 [2026-10-20 00:54:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.7587 [2026-10-20 00:54:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.6070 [2026-10-20 00:54:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.2806 [2026-10-20 00:54:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.6033 [2026-10-20 00:54:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.2281 [2026-10-20 00:55:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3628 [2026-10-20 00:55:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.5728 [2026-10-20 00:55:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4733 [2026-10-20 00:55:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4419 [2026-10-20 00:55:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.1751 [2026-10-20 00:55:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.7210 [2026-10-20 00:55:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.5541 [2026-10-20 00:55:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.6955 [2026-10-20 00:55:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.9316 [2026-10-20 00:55:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.8119 [2026-10-20 00:55:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.6892 [2026-10-20 00:55:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.8001 [2026-10-20 00:56:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.6445 [2026-10-20 00:56:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.7781 [2026-10-20 00:56:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.6224 [2026-10-20 00:56:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.9761 [2026-10-20 00:56:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.8555 [2026-10-20 00:56:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.9752 [2026-10-20 00:56:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.6482 [2026-10-20 00:56:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.1007 [2026-10-20 00:56:40][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,563 | MFU: 1.8% [2026-10-20 00:56:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.6685 [2026-10-20 00:56:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.6101 [2026-10-20 00:56:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.5940 [2026-10-20 00:56:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.7039 [2026-10-20 00:56:44][INFO][training.pretrain_engine] [Ph2|Step 157] loss=7.8557 | ppl=2580.48 | aux=0.02516 | lr=1.23e-04 | gnorm=0.703 | tok/s=29,271 | MFU=7.0% | step=143292ms | tokens=0.172B | mem=4.2/32.7GB | cost=$82.45/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 21:39 (140.7h) | Phase3: SKIPPED | SFT: Oct 26 02:24 (145.5h) | DPO (Final): Oct 26 02:58 (146.0h) [2026-10-20 00:56:44][INFO][training.pretrain_engine] Starting global step 158 (accumulating 32 micro-batches)... [2026-10-20 00:56:44][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 00:57:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.0322 [2026-10-20 00:57:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.7722 [2026-10-20 00:57:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.3856 [2026-10-20 00:57:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.7148 [2026-10-20 00:57:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.9587 [2026-10-20 00:57:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.8944 [2026-10-20 00:57:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.7717 [2026-10-20 00:57:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.9862 [2026-10-20 00:57:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.1393 [2026-10-20 00:57:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.3323 [2026-10-20 00:57:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.2617 [2026-10-20 00:57:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.4925 [2026-10-20 00:57:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.5638 [2026-10-20 00:57:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.5479 [2026-10-20 00:57:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.8132 [2026-10-20 00:57:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.1304 [2026-10-20 00:58:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.8274 [2026-10-20 00:58:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.4947 [2026-10-20 00:58:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.7599 [2026-10-20 00:58:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.1711 [2026-10-20 00:58:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.7251 [2026-10-20 00:58:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.6203 [2026-10-20 00:58:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.9206 [2026-10-20 00:58:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.6470 [2026-10-20 00:58:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.0784 [2026-10-20 00:58:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.2001 [2026-10-20 00:58:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.7974 [2026-10-20 00:58:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.6820 [2026-10-20 00:59:03][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,536 | MFU: 1.8% [2026-10-20 00:59:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.6839 [2026-10-20 00:59:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.8404 [2026-10-20 00:59:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.5884 [2026-10-20 00:59:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4594 [2026-10-20 00:59:08][INFO][training.pretrain_engine] [Ph2|Step 158] loss=7.8901 | ppl=2670.81 | aux=0.02518 | lr=1.23e-04 | gnorm=0.676 | tok/s=29,270 | MFU=7.0% | step=143297ms | tokens=0.173B | mem=4.2/32.7GB | cost=$82.97/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 21:41 (140.7h) | Phase3: SKIPPED | SFT: Oct 26 02:26 (145.5h) | DPO (Final): Oct 26 03:00 (146.0h) [2026-10-20 00:59:08][INFO][training.pretrain_engine] Starting global step 159 (accumulating 32 micro-batches)... [2026-10-20 00:59:08][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 00:59:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.6211 [2026-10-20 00:59:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.0288 [2026-10-20 00:59:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.8363 [2026-10-20 00:59:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.6350 [2026-10-20 00:59:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.6246 [2026-10-20 00:59:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.8052 [2026-10-20 00:59:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.7977 [2026-10-20 00:59:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4694 [2026-10-20 01:00:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.7259 [2026-10-20 01:00:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.1794 [2026-10-20 01:00:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.7698 [2026-10-20 01:00:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.6568 [2026-10-20 01:00:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.5798 [2026-10-20 01:00:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.8565 [2026-10-20 01:00:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.6272 [2026-10-20 01:00:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.9288 [2026-10-20 01:00:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.8868 [2026-10-20 01:00:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.7876 [2026-10-20 01:00:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.1041 [2026-10-20 01:00:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.7135 [2026-10-20 01:00:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.8153 [2026-10-20 01:00:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.7531 [2026-10-20 01:00:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.4807 [2026-10-20 01:00:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.7269 [2026-10-20 01:01:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.6325 [2026-10-20 01:01:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.3742 [2026-10-20 01:01:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.2381 [2026-10-20 01:01:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.6199 [2026-10-20 01:01:27][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,558 | MFU: 1.8% [2026-10-20 01:01:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.5849 [2026-10-20 01:01:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4922 [2026-10-20 01:01:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.6837 [2026-10-20 01:01:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.6138 [2026-10-20 01:01:32][INFO][training.pretrain_engine] [Ph2|Step 159] loss=7.8965 | ppl=2687.87 | aux=0.02518 | lr=1.24e-04 | gnorm=0.863 | tok/s=29,269 | MFU=7.0% | step=143300ms | tokens=0.174B | mem=4.2/32.7GB | cost=$83.50/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 21:43 (140.7h) | Phase3: SKIPPED | SFT: Oct 26 02:28 (145.4h) | DPO (Final): Oct 26 03:02 (146.0h) [2026-10-20 01:01:32][INFO][training.pretrain_engine] Starting global step 160 (accumulating 32 micro-batches)... [2026-10-20 01:01:32][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 01:01:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.9103 [2026-10-20 01:01:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.5328 [2026-10-20 01:01:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.6148 [2026-10-20 01:01:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.4349 [2026-10-20 01:02:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.5602 [2026-10-20 01:02:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.6334 [2026-10-20 01:02:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.5818 [2026-10-20 01:02:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.6686 [2026-10-20 01:02:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.8171 [2026-10-20 01:02:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.9836 [2026-10-20 01:02:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.4137 [2026-10-20 01:02:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.2353 [2026-10-20 01:02:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.7185 [2026-10-20 01:02:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4919 [2026-10-20 01:02:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.3613 [2026-10-20 01:02:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.9071 [2026-10-20 01:03:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2722 [2026-10-20 01:03:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3229 [2026-10-20 01:03:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.4935 [2026-10-20 01:03:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.5694 [2026-10-20 01:03:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.7429 [2026-10-20 01:03:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.5556 [2026-10-20 01:03:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.7181 [2026-10-20 01:03:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.5276 [2026-10-20 01:03:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.7438 [2026-10-20 01:03:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.6365 [2026-10-20 01:03:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.8640 [2026-10-20 01:03:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.5609 [2026-10-20 01:03:50][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,556 | MFU: 1.8% [2026-10-20 01:03:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.7492 [2026-10-20 01:03:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.8792 [2026-10-20 01:03:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.0327 [2026-10-20 01:03:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.9806 [2026-10-20 01:03:57][INFO][training.pretrain_engine] [Ph2|Step 160] loss=7.7845 | ppl=2403.16 | aux=0.02516 | lr=1.25e-04 | gnorm=1.172 | tok/s=29,266 | MFU=7.0% | step=143317ms | tokens=0.175B | mem=4.2/32.7GB | cost=$84.03/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 21:46 (140.7h) | Phase3: SKIPPED | SFT: Oct 26 02:31 (145.5h) | DPO (Final): Oct 26 03:05 (146.0h) [2026-10-20 01:03:57][INFO][training.pretrain_engine] Starting global step 161 (accumulating 32 micro-batches)... [2026-10-20 01:03:57][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 01:04:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.8973 [2026-10-20 01:04:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.1328 [2026-10-20 01:04:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.0084 [2026-10-20 01:04:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.7357 [2026-10-20 01:04:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.8921 [2026-10-20 01:04:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4280 [2026-10-20 01:04:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.1667 [2026-10-20 01:04:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.5411 [2026-10-20 01:04:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.1377 [2026-10-20 01:04:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.1111 [2026-10-20 01:04:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.2939 [2026-10-20 01:04:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.9440 [2026-10-20 01:05:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.6709 [2026-10-20 01:05:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.7280 [2026-10-20 01:05:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.9733 [2026-10-20 01:05:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.0081 [2026-10-20 01:05:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.8700 [2026-10-20 01:05:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.4657 [2026-10-20 01:05:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.2006 [2026-10-20 01:05:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.6045 [2026-10-20 01:05:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.9036 [2026-10-20 01:05:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.0221 [2026-10-20 01:05:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.9623 [2026-10-20 01:05:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.6522 [2026-10-20 01:06:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.5574 [2026-10-20 01:06:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.6201 [2026-10-20 01:06:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.2275 [2026-10-20 01:06:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.6640 [2026-10-20 01:06:15][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,555 | MFU: 1.8% [2026-10-20 01:06:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.7301 [2026-10-20 01:06:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.5686 [2026-10-20 01:06:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3930 [2026-10-20 01:06:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4653 [2026-10-20 01:06:21][INFO][training.pretrain_engine] [Ph2|Step 161] loss=7.8689 | ppl=2614.57 | aux=0.02519 | lr=1.25e-04 | gnorm=0.898 | tok/s=29,263 | MFU=7.0% | step=143332ms | tokens=0.176B | mem=4.2/32.7GB | cost=$84.56/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 21:49 (140.7h) | Phase3: SKIPPED | SFT: Oct 26 02:34 (145.5h) | DPO (Final): Oct 26 03:08 (146.0h) [2026-10-20 01:06:21][INFO][training.pretrain_engine] Starting global step 162 (accumulating 32 micro-batches)... [2026-10-20 01:06:21][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 01:06:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.4545 [2026-10-20 01:06:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2074 [2026-10-20 01:06:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.4182 [2026-10-20 01:06:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.3334 [2026-10-20 01:06:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.5334 [2026-10-20 01:06:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.7434 [2026-10-20 01:06:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.6754 [2026-10-20 01:06:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.6809 [2026-10-20 01:07:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.6147 [2026-10-20 01:07:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.7996 [2026-10-20 01:07:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.5279 [2026-10-20 01:07:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.5572 [2026-10-20 01:07:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.5518 [2026-10-20 01:07:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.9047 [2026-10-20 01:07:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.9809 [2026-10-20 01:07:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.7430 [2026-10-20 01:07:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.2064 [2026-10-20 01:07:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.2444 [2026-10-20 01:07:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.2448 [2026-10-20 01:07:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.6158 [2026-10-20 01:08:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.5970 [2026-10-20 01:08:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.6883 [2026-10-20 01:08:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.4255 [2026-10-20 01:08:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.1447 [2026-10-20 01:08:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.7256 [2026-10-20 01:08:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3089 [2026-10-20 01:08:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.7537 [2026-10-20 01:08:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.6118 [2026-10-20 01:08:40][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,560 | MFU: 1.8% [2026-10-20 01:08:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.8498 [2026-10-20 01:08:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.7115 [2026-10-20 01:08:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.3010 [2026-10-20 01:08:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.1022 [2026-10-20 01:08:46][INFO][training.pretrain_engine] [Ph2|Step 162] loss=7.8314 | ppl=2518.56 | aux=0.02523 | lr=1.26e-04 | gnorm=0.598 | tok/s=29,259 | MFU=7.0% | step=143352ms | tokens=0.177B | mem=4.2/32.7GB | cost=$85.09/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 21:52 (140.7h) | Phase3: SKIPPED | SFT: Oct 26 02:37 (145.5h) | DPO (Final): Oct 26 03:11 (146.0h) [2026-10-20 01:08:49][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step162 | Tokens: 0.177B | Spent: $85.10 | Remaining: $314.90 [2026-10-20 01:08:49][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step162 [2026-10-20 01:08:49][INFO][training.pretrain_engine] Starting global step 163 (accumulating 32 micro-batches)... [2026-10-20 01:08:49][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 01:08:49][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 01:08:49][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 01:08:49][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 01:08:49][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step162 to Phase-Technologies/forge-500m-pretrain (path: phase2_step162)... [2026-10-20 01:08:49][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 01:08:59][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 01:08:59][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step162 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 01:09:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.4503 [2026-10-20 01:09:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.7019 [2026-10-20 01:09:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.5511 [2026-10-20 01:09:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.8293 [2026-10-20 01:09:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.2134 [2026-10-20 01:09:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.3312 [2026-10-20 01:09:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.0497 [2026-10-20 01:09:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.4714 [2026-10-20 01:09:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.0486 [2026-10-20 01:09:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4344 [2026-10-20 01:09:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.5875 [2026-10-20 01:09:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.5900 [2026-10-20 01:09:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.9192 [2026-10-20 01:09:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.9567 [2026-10-20 01:09:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.9004 [2026-10-20 01:10:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.9198 [2026-10-20 01:10:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.1828 [2026-10-20 01:10:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.6463 [2026-10-20 01:10:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.4007 [2026-10-20 01:10:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.4097 [2026-10-20 01:10:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.5996 [2026-10-20 01:10:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.8603 [2026-10-20 01:10:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.8175 [2026-10-20 01:10:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.9763 [2026-10-20 01:10:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.9563 [2026-10-20 01:10:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.4122 [2026-10-20 01:10:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.9426 [2026-10-20 01:10:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.1322 [2026-10-20 01:11:07][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,578 | MFU: 1.8% [2026-10-20 01:11:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.8797 [2026-10-20 01:11:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.1480 [2026-10-20 01:11:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.5647 [2026-10-20 01:11:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.5443 [2026-10-20 01:11:12][INFO][training.pretrain_engine] [Ph2|Step 163] loss=7.8480 | ppl=2560.56 | aux=0.02518 | lr=1.26e-04 | gnorm=0.785 | tok/s=29,259 | MFU=7.0% | step=143350ms | tokens=0.178B | mem=4.2/32.7GB | cost=$85.62/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 21:53 (140.7h) | Phase3: SKIPPED | SFT: Oct 26 02:38 (145.5h) | DPO (Final): Oct 26 03:12 (146.0h) [2026-10-20 01:11:12][INFO][training.pretrain_engine] Starting global step 164 (accumulating 32 micro-batches)... [2026-10-20 01:11:12][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 01:11:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.4865 [2026-10-20 01:11:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.6294 [2026-10-20 01:11:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.9155 [2026-10-20 01:11:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.7873 [2026-10-20 01:11:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.9101 [2026-10-20 01:11:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.6692 [2026-10-20 01:11:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.8549 [2026-10-20 01:11:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.5512 [2026-10-20 01:12:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.1094 [2026-10-20 01:12:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.1051 [2026-10-20 01:12:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.1025 [2026-10-20 01:12:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.1337 [2026-10-20 01:12:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.6338 [2026-10-20 01:12:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.7273 [2026-10-20 01:12:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.8176 [2026-10-20 01:12:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.9197 [2026-10-20 01:12:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.1317 [2026-10-20 01:12:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.1194 [2026-10-20 01:12:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.8841 [2026-10-20 01:12:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.5625 [2026-10-20 01:12:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.9487 [2026-10-20 01:12:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.9543 [2026-10-20 01:12:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.2018 [2026-10-20 01:12:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.9620 [2026-10-20 01:13:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.7541 [2026-10-20 01:13:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.1058 [2026-10-20 01:13:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.9276 [2026-10-20 01:13:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.7713 [2026-10-20 01:13:30][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,579 | MFU: 1.8% [2026-10-20 01:13:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.5991 [2026-10-20 01:13:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.6750 [2026-10-20 01:13:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.6291 [2026-10-20 01:13:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.9669 [2026-10-20 01:13:35][INFO][training.pretrain_engine] [Ph2|Step 164] loss=7.8638 | ppl=2601.36 | aux=0.02517 | lr=1.27e-04 | gnorm=0.922 | tok/s=29,258 | MFU=7.0% | step=143353ms | tokens=0.179B | mem=4.2/32.7GB | cost=$86.14/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 21:55 (140.7h) | Phase3: SKIPPED | SFT: Oct 26 02:40 (145.5h) | DPO (Final): Oct 26 03:14 (146.0h) [2026-10-20 01:13:35][INFO][training.pretrain_engine] Starting global step 165 (accumulating 32 micro-batches)... [2026-10-20 01:13:35][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 01:13:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.8185 [2026-10-20 01:13:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.0968 [2026-10-20 01:13:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.8919 [2026-10-20 01:13:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.5240 [2026-10-20 01:14:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.8590 [2026-10-20 01:14:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.5814 [2026-10-20 01:14:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4888 [2026-10-20 01:14:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.0995 [2026-10-20 01:14:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4383 [2026-10-20 01:14:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4130 [2026-10-20 01:14:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.8393 [2026-10-20 01:14:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.5201 [2026-10-20 01:14:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4067 [2026-10-20 01:14:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.5620 [2026-10-20 01:14:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.0683 [2026-10-20 01:14:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.7754 [2026-10-20 01:15:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.2052 [2026-10-20 01:15:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.1236 [2026-10-20 01:15:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.9312 [2026-10-20 01:15:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.0990 [2026-10-20 01:15:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.6274 [2026-10-20 01:15:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.9292 [2026-10-20 01:15:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.7063 [2026-10-20 01:15:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.7145 [2026-10-20 01:15:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.6731 [2026-10-20 01:15:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.1893 [2026-10-20 01:15:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.8446 [2026-10-20 01:15:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.7831 [2026-10-20 01:15:53][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,576 | MFU: 1.8% [2026-10-20 01:15:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.6652 [2026-10-20 01:15:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.6207 [2026-10-20 01:15:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3037 [2026-10-20 01:15:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.5806 [2026-10-20 01:15:58][INFO][training.pretrain_engine] [Ph2|Step 165] loss=7.8343 | ppl=2525.88 | aux=0.02520 | lr=1.27e-04 | gnorm=0.785 | tok/s=29,260 | MFU=7.0% | step=143346ms | tokens=0.180B | mem=4.2/32.7GB | cost=$86.67/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 21:57 (140.7h) | Phase3: SKIPPED | SFT: Oct 26 02:42 (145.4h) | DPO (Final): Oct 26 03:16 (146.0h) [2026-10-20 01:15:58][INFO][training.pretrain_engine] Starting global step 166 (accumulating 32 micro-batches)... [2026-10-20 01:15:58][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 01:16:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.8879 [2026-10-20 01:16:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.7081 [2026-10-20 01:16:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.7420 [2026-10-20 01:16:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.2528 [2026-10-20 01:16:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.8825 [2026-10-20 01:16:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.7906 [2026-10-20 01:16:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.9368 [2026-10-20 01:16:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.7655 [2026-10-20 01:16:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.7842 [2026-10-20 01:16:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.6383 [2026-10-20 01:16:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.6281 [2026-10-20 01:16:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.5653 [2026-10-20 01:17:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.5764 [2026-10-20 01:17:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.1118 [2026-10-20 01:17:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2803 [2026-10-20 01:17:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.5085 [2026-10-20 01:17:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.9211 [2026-10-20 01:17:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.7900 [2026-10-20 01:17:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.8057 [2026-10-20 01:17:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.8657 [2026-10-20 01:17:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.9336 [2026-10-20 01:17:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.0891 [2026-10-20 01:17:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.9386 [2026-10-20 01:17:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.1783 [2026-10-20 01:18:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.8221 [2026-10-20 01:18:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.7721 [2026-10-20 01:18:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.8233 [2026-10-20 01:18:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.4661 [2026-10-20 01:18:16][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,577 | MFU: 1.8% [2026-10-20 01:18:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.1768 [2026-10-20 01:18:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.7009 [2026-10-20 01:18:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.2383 [2026-10-20 01:18:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.8527 [2026-10-20 01:18:21][INFO][training.pretrain_engine] [Ph2|Step 166] loss=7.7545 | ppl=2332.13 | aux=0.02516 | lr=1.28e-04 | gnorm=0.574 | tok/s=29,263 | MFU=7.0% | step=143333ms | tokens=0.181B | mem=4.2/32.7GB | cost=$87.19/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 21:58 (140.7h) | Phase3: SKIPPED | SFT: Oct 26 02:43 (145.4h) | DPO (Final): Oct 26 03:17 (146.0h) [2026-10-20 01:18:21][INFO][training.pretrain_engine] Starting global step 167 (accumulating 32 micro-batches)... [2026-10-20 01:18:21][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 01:18:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.5221 [2026-10-20 01:18:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.9021 [2026-10-20 01:18:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.7132 [2026-10-20 01:18:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.4445 [2026-10-20 01:18:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.7939 [2026-10-20 01:18:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.2524 [2026-10-20 01:18:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.8916 [2026-10-20 01:18:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.1712 [2026-10-20 01:19:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.0613 [2026-10-20 01:19:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.1170 [2026-10-20 01:19:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3781 [2026-10-20 01:19:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.7815 [2026-10-20 01:19:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4064 [2026-10-20 01:19:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4241 [2026-10-20 01:19:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.3653 [2026-10-20 01:19:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.6512 [2026-10-20 01:19:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2825 [2026-10-20 01:19:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.8670 [2026-10-20 01:19:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.0332 [2026-10-20 01:19:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.9668 [2026-10-20 01:20:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.6368 [2026-10-20 01:20:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.3163 [2026-10-20 01:20:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.3252 [2026-10-20 01:20:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.6132 [2026-10-20 01:20:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.7578 [2026-10-20 01:20:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.6615 [2026-10-20 01:20:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.7008 [2026-10-20 01:20:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.6504 [2026-10-20 01:20:39][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,583 | MFU: 1.8% [2026-10-20 01:20:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.0600 [2026-10-20 01:20:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.9324 [2026-10-20 01:20:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.9210 [2026-10-20 01:20:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.2058 [2026-10-20 01:20:44][INFO][training.pretrain_engine] [Ph2|Step 167] loss=7.7161 | ppl=2244.21 | aux=0.02514 | lr=1.29e-04 | gnorm=0.660 | tok/s=29,264 | MFU=7.0% | step=143328ms | tokens=0.182B | mem=4.2/32.7GB | cost=$87.71/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 21:59 (140.7h) | Phase3: SKIPPED | SFT: Oct 26 02:44 (145.4h) | DPO (Final): Oct 26 03:18 (146.0h) [2026-10-20 01:20:44][INFO][training.pretrain_engine] Starting global step 168 (accumulating 32 micro-batches)... [2026-10-20 01:20:44][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 01:21:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.5218 [2026-10-20 01:21:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.4976 [2026-10-20 01:21:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2859 [2026-10-20 01:21:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.3535 [2026-10-20 01:21:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.6330 [2026-10-20 01:21:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=9.1949 [2026-10-20 01:21:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.2369 [2026-10-20 01:21:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.3852 [2026-10-20 01:21:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.8541 [2026-10-20 01:21:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3611 [2026-10-20 01:21:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4479 [2026-10-20 01:21:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4736 [2026-10-20 01:21:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4555 [2026-10-20 01:21:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.8705 [2026-10-20 01:21:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.5566 [2026-10-20 01:21:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.5505 [2026-10-20 01:22:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.6350 [2026-10-20 01:22:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.7260 [2026-10-20 01:22:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.6729 [2026-10-20 01:22:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.7647 [2026-10-20 01:22:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.3275 [2026-10-20 01:22:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.4030 [2026-10-20 01:22:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.6810 [2026-10-20 01:22:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.4461 [2026-10-20 01:22:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.4039 [2026-10-20 01:22:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.4417 [2026-10-20 01:22:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.6256 [2026-10-20 01:22:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.6178 [2026-10-20 01:23:02][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,572 | MFU: 1.8% [2026-10-20 01:23:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.5280 [2026-10-20 01:23:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.3257 [2026-10-20 01:23:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2630 [2026-10-20 01:23:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.1271 [2026-10-20 01:23:07][INFO][training.pretrain_engine] [Ph2|Step 168] loss=7.7462 | ppl=2312.84 | aux=0.02517 | lr=1.29e-04 | gnorm=0.777 | tok/s=29,264 | MFU=7.0% | step=143327ms | tokens=0.184B | mem=4.2/32.7GB | cost=$88.23/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 22:01 (140.6h) | Phase3: SKIPPED | SFT: Oct 26 02:46 (145.4h) | DPO (Final): Oct 26 03:20 (146.0h) [2026-10-20 01:23:07][INFO][training.pretrain_engine] Starting global step 169 (accumulating 32 micro-batches)... [2026-10-20 01:23:07][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 01:23:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.4692 [2026-10-20 01:23:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.5257 [2026-10-20 01:23:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.4703 [2026-10-20 01:23:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.3759 [2026-10-20 01:23:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.3270 [2026-10-20 01:23:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.5191 [2026-10-20 01:23:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.0903 [2026-10-20 01:23:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.7597 [2026-10-20 01:24:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.5884 [2026-10-20 01:24:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3521 [2026-10-20 01:24:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.5408 [2026-10-20 01:24:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4772 [2026-10-20 01:24:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.6893 [2026-10-20 01:24:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4538 [2026-10-20 01:24:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4767 [2026-10-20 01:24:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.7016 [2026-10-20 01:24:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.7767 [2026-10-20 01:24:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.5458 [2026-10-20 01:24:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3008 [2026-10-20 01:24:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.7120 [2026-10-20 01:24:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.0189 [2026-10-20 01:24:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2626 [2026-10-20 01:24:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.8386 [2026-10-20 01:24:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.7971 [2026-10-20 01:25:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.3436 [2026-10-20 01:25:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1799 [2026-10-20 01:25:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.9811 [2026-10-20 01:25:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.3266 [2026-10-20 01:25:25][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,570 | MFU: 1.8% [2026-10-20 01:25:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3852 [2026-10-20 01:25:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3172 [2026-10-20 01:25:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.5817 [2026-10-20 01:25:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.1551 [2026-10-20 01:25:30][INFO][training.pretrain_engine] [Ph2|Step 169] loss=7.6545 | ppl=2110.16 | aux=0.02513 | lr=1.30e-04 | gnorm=0.820 | tok/s=29,264 | MFU=7.0% | step=143325ms | tokens=0.185B | mem=4.2/32.7GB | cost=$88.76/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 22:03 (140.6h) | Phase3: SKIPPED | SFT: Oct 26 02:48 (145.4h) | DPO (Final): Oct 26 03:22 (145.9h) [2026-10-20 01:25:30][INFO][training.pretrain_engine] Starting global step 170 (accumulating 32 micro-batches)... [2026-10-20 01:25:30][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 01:25:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.8103 [2026-10-20 01:25:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.4648 [2026-10-20 01:25:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.8681 [2026-10-20 01:25:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.7127 [2026-10-20 01:26:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.3922 [2026-10-20 01:26:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4625 [2026-10-20 01:26:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.3462 [2026-10-20 01:26:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.6567 [2026-10-20 01:26:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.5058 [2026-10-20 01:26:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.7295 [2026-10-20 01:26:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.2312 [2026-10-20 01:26:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.5449 [2026-10-20 01:26:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.8551 [2026-10-20 01:26:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0679 [2026-10-20 01:26:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.5802 [2026-10-20 01:26:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.8085 [2026-10-20 01:26:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.8826 [2026-10-20 01:26:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.7370 [2026-10-20 01:26:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.6099 [2026-10-20 01:26:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.9837 [2026-10-20 01:27:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.5481 [2026-10-20 01:27:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.6312 [2026-10-20 01:27:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.1306 [2026-10-20 01:27:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.1236 [2026-10-20 01:27:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.6816 [2026-10-20 01:27:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.8231 [2026-10-20 01:27:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.5528 [2026-10-20 01:27:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.4616 [2026-10-20 01:27:48][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,585 | MFU: 1.8% [2026-10-20 01:27:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.0412 [2026-10-20 01:27:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.1734 [2026-10-20 01:27:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.0814 [2026-10-20 01:27:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.5870 [2026-10-20 01:27:53][INFO][training.pretrain_engine] [Ph2|Step 170] loss=7.7795 | ppl=2390.96 | aux=0.02514 | lr=1.30e-04 | gnorm=0.789 | tok/s=29,265 | MFU=7.0% | step=143322ms | tokens=0.186B | mem=4.2/32.7GB | cost=$89.28/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 22:04 (140.6h) | Phase3: SKIPPED | SFT: Oct 26 02:49 (145.4h) | DPO (Final): Oct 26 03:23 (145.9h) [2026-10-20 01:27:53][INFO][training.pretrain_engine] Starting global step 171 (accumulating 32 micro-batches)... [2026-10-20 01:27:53][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 01:28:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.4488 [2026-10-20 01:28:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.2304 [2026-10-20 01:28:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.5316 [2026-10-20 01:28:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.4408 [2026-10-20 01:28:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.6033 [2026-10-20 01:28:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.3866 [2026-10-20 01:28:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.5907 [2026-10-20 01:28:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.9788 [2026-10-20 01:28:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.6808 [2026-10-20 01:28:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.6201 [2026-10-20 01:28:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4544 [2026-10-20 01:28:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4277 [2026-10-20 01:29:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.3635 [2026-10-20 01:29:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.3042 [2026-10-20 01:29:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4755 [2026-10-20 01:29:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4890 [2026-10-20 01:29:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.8532 [2026-10-20 01:29:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.1310 [2026-10-20 01:29:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.7866 [2026-10-20 01:29:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.0503 [2026-10-20 01:29:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.6203 [2026-10-20 01:29:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.0028 [2026-10-20 01:29:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.6355 [2026-10-20 01:29:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.5605 [2026-10-20 01:29:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.6829 [2026-10-20 01:29:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.4939 [2026-10-20 01:29:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.5344 [2026-10-20 01:29:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.7089 [2026-10-20 01:30:12][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,563 | MFU: 1.8% [2026-10-20 01:30:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.6450 [2026-10-20 01:30:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9731 [2026-10-20 01:30:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.1860 [2026-10-20 01:30:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3395 [2026-10-20 01:30:16][INFO][training.pretrain_engine] [Ph2|Step 171] loss=7.6204 | ppl=2039.32 | aux=0.02517 | lr=1.31e-04 | gnorm=0.891 | tok/s=29,265 | MFU=7.0% | step=143323ms | tokens=0.187B | mem=4.2/32.7GB | cost=$89.80/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 22:06 (140.6h) | Phase3: SKIPPED | SFT: Oct 26 02:51 (145.4h) | DPO (Final): Oct 26 03:25 (145.9h) [2026-10-20 01:30:20][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step171 | Tokens: 0.187B | Spent: $89.82 | Remaining: $310.18 [2026-10-20 01:30:20][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step171 [2026-10-20 01:30:20][INFO][training.pretrain_engine] Starting global step 172 (accumulating 32 micro-batches)... [2026-10-20 01:30:20][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 01:30:20][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 01:30:20][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 01:30:20][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 01:30:20][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step171 to Phase-Technologies/forge-500m-pretrain (path: phase2_step171)... [2026-10-20 01:30:20][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 01:30:30][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 01:30:30][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step171 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 01:30:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.6863 [2026-10-20 01:30:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2478 [2026-10-20 01:30:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.3339 [2026-10-20 01:30:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.3154 [2026-10-20 01:30:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2288 [2026-10-20 01:30:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.6515 [2026-10-20 01:30:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4379 [2026-10-20 01:30:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.5859 [2026-10-20 01:31:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.0900 [2026-10-20 01:31:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.1220 [2026-10-20 01:31:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.7915 [2026-10-20 01:31:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.8863 [2026-10-20 01:31:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.7094 [2026-10-20 01:31:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.6619 [2026-10-20 01:31:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4104 [2026-10-20 01:31:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.6951 [2026-10-20 01:31:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2895 [2026-10-20 01:31:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.5174 [2026-10-20 01:31:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1537 [2026-10-20 01:31:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.6492 [2026-10-20 01:32:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.3247 [2026-10-20 01:32:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.7657 [2026-10-20 01:32:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.8218 [2026-10-20 01:32:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.4923 [2026-10-20 01:32:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.4099 [2026-10-20 01:32:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.6361 [2026-10-20 01:32:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.8121 [2026-10-20 01:32:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.7138 [2026-10-20 01:32:39][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,557 | MFU: 1.8% [2026-10-20 01:32:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3802 [2026-10-20 01:32:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.6684 [2026-10-20 01:32:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.6275 [2026-10-20 01:32:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.7427 [2026-10-20 01:32:43][INFO][training.pretrain_engine] [Ph2|Step 172] loss=7.7049 | ppl=2219.17 | aux=0.02518 | lr=1.31e-04 | gnorm=0.707 | tok/s=29,265 | MFU=7.0% | step=143322ms | tokens=0.188B | mem=4.2/32.7GB | cost=$90.34/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 22:08 (140.6h) | Phase3: SKIPPED | SFT: Oct 26 02:53 (145.3h) | DPO (Final): Oct 26 03:27 (145.9h) [2026-10-20 01:32:43][INFO][training.pretrain_engine] Starting global step 173 (accumulating 32 micro-batches)... [2026-10-20 01:32:43][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 01:33:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.4480 [2026-10-20 01:33:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.3582 [2026-10-20 01:33:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.5647 [2026-10-20 01:33:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2656 [2026-10-20 01:33:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.3259 [2026-10-20 01:33:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.1432 [2026-10-20 01:33:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.1468 [2026-10-20 01:33:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.9510 [2026-10-20 01:33:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4244 [2026-10-20 01:33:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4927 [2026-10-20 01:33:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4129 [2026-10-20 01:33:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.8493 [2026-10-20 01:33:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.7591 [2026-10-20 01:33:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.7202 [2026-10-20 01:33:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.5318 [2026-10-20 01:33:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2381 [2026-10-20 01:34:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.8296 [2026-10-20 01:34:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.2160 [2026-10-20 01:34:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.1175 [2026-10-20 01:34:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.5074 [2026-10-20 01:34:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.4119 [2026-10-20 01:34:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.5498 [2026-10-20 01:34:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.3858 [2026-10-20 01:34:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.5460 [2026-10-20 01:34:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.1510 [2026-10-20 01:34:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.6115 [2026-10-20 01:34:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.0149 [2026-10-20 01:34:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.5009 [2026-10-20 01:35:02][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,570 | MFU: 1.8% [2026-10-20 01:35:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4552 [2026-10-20 01:35:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2896 [2026-10-20 01:35:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3525 [2026-10-20 01:35:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3947 [2026-10-20 01:35:06][INFO][training.pretrain_engine] [Ph2|Step 173] loss=7.6743 | ppl=2152.40 | aux=0.02516 | lr=1.32e-04 | gnorm=0.684 | tok/s=29,264 | MFU=7.0% | step=143328ms | tokens=0.189B | mem=4.2/32.7GB | cost=$90.86/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 22:10 (140.6h) | Phase3: SKIPPED | SFT: Oct 26 02:55 (145.3h) | DPO (Final): Oct 26 03:29 (145.9h) [2026-10-20 01:35:06][INFO][training.pretrain_engine] Starting global step 174 (accumulating 32 micro-batches)... [2026-10-20 01:35:06][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 01:35:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.5518 [2026-10-20 01:35:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.4808 [2026-10-20 01:35:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.5496 [2026-10-20 01:35:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.6415 [2026-10-20 01:35:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4165 [2026-10-20 01:35:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.3894 [2026-10-20 01:35:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2985 [2026-10-20 01:35:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.3930 [2026-10-20 01:36:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.2417 [2026-10-20 01:36:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.1090 [2026-10-20 01:36:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4662 [2026-10-20 01:36:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1514 [2026-10-20 01:36:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.0157 [2026-10-20 01:36:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.9226 [2026-10-20 01:36:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.5758 [2026-10-20 01:36:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.3623 [2026-10-20 01:36:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.5615 [2026-10-20 01:36:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.4296 [2026-10-20 01:36:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3899 [2026-10-20 01:36:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0737 [2026-10-20 01:36:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.5662 [2026-10-20 01:36:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2319 [2026-10-20 01:36:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.6512 [2026-10-20 01:36:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2552 [2026-10-20 01:37:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.4941 [2026-10-20 01:37:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.8826 [2026-10-20 01:37:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.4368 [2026-10-20 01:37:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2542 [2026-10-20 01:37:24][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,584 | MFU: 1.8% [2026-10-20 01:37:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4957 [2026-10-20 01:37:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.5114 [2026-10-20 01:37:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4587 [2026-10-20 01:37:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4933 [2026-10-20 01:37:29][INFO][training.pretrain_engine] [Ph2|Step 174] loss=7.6401 | ppl=2079.94 | aux=0.02515 | lr=1.32e-04 | gnorm=0.977 | tok/s=29,264 | MFU=7.0% | step=143327ms | tokens=0.190B | mem=4.2/32.7GB | cost=$91.39/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 22:12 (140.6h) | Phase3: SKIPPED | SFT: Oct 26 02:57 (145.3h) | DPO (Final): Oct 26 03:31 (145.9h) [2026-10-20 01:37:29][INFO][training.pretrain_engine] Starting global step 175 (accumulating 32 micro-batches)... [2026-10-20 01:37:29][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 01:37:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.9246 [2026-10-20 01:37:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.1757 [2026-10-20 01:37:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.5799 [2026-10-20 01:37:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.2015 [2026-10-20 01:38:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.8354 [2026-10-20 01:38:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.7188 [2026-10-20 01:38:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.5158 [2026-10-20 01:38:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.8102 [2026-10-20 01:38:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4573 [2026-10-20 01:38:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.5149 [2026-10-20 01:38:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3643 [2026-10-20 01:38:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4057 [2026-10-20 01:38:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4194 [2026-10-20 01:38:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.7239 [2026-10-20 01:38:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1848 [2026-10-20 01:38:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4161 [2026-10-20 01:38:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.4590 [2026-10-20 01:38:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.1032 [2026-10-20 01:38:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.4736 [2026-10-20 01:38:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.6545 [2026-10-20 01:39:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.2851 [2026-10-20 01:39:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.1769 [2026-10-20 01:39:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.5554 [2026-10-20 01:39:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.1671 [2026-10-20 01:39:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.9126 [2026-10-20 01:39:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.6780 [2026-10-20 01:39:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.9805 [2026-10-20 01:39:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.8037 [2026-10-20 01:39:48][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,571 | MFU: 1.8% [2026-10-20 01:39:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.6517 [2026-10-20 01:39:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.8120 [2026-10-20 01:39:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.8129 [2026-10-20 01:39:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2534 [2026-10-20 01:39:52][INFO][training.pretrain_engine] [Ph2|Step 175] loss=7.5923 | ppl=1982.94 | aux=0.02515 | lr=1.33e-04 | gnorm=1.281 | tok/s=29,264 | MFU=7.0% | step=143325ms | tokens=0.191B | mem=4.2/32.7GB | cost=$91.91/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 22:14 (140.6h) | Phase3: SKIPPED | SFT: Oct 26 02:58 (145.3h) | DPO (Final): Oct 26 03:32 (145.9h) [2026-10-20 01:39:52][INFO][training.pretrain_engine] Starting global step 176 (accumulating 32 micro-batches)... [2026-10-20 01:39:52][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 01:40:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.5348 [2026-10-20 01:40:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.8254 [2026-10-20 01:40:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.6108 [2026-10-20 01:40:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.7347 [2026-10-20 01:40:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.5956 [2026-10-20 01:40:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.5918 [2026-10-20 01:40:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.5483 [2026-10-20 01:40:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.5899 [2026-10-20 01:40:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.1320 [2026-10-20 01:40:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.6261 [2026-10-20 01:40:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.2955 [2026-10-20 01:40:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.3333 [2026-10-20 01:41:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.7601 [2026-10-20 01:41:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.7977 [2026-10-20 01:41:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.5600 [2026-10-20 01:41:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.6426 [2026-10-20 01:41:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.5907 [2026-10-20 01:41:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.7080 [2026-10-20 01:41:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.6702 [2026-10-20 01:41:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3360 [2026-10-20 01:41:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.5822 [2026-10-20 01:41:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.4185 [2026-10-20 01:41:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.1502 [2026-10-20 01:41:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.4895 [2026-10-20 01:41:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.5820 [2026-10-20 01:41:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2167 [2026-10-20 01:41:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1610 [2026-10-20 01:41:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.7172 [2026-10-20 01:42:11][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,564 | MFU: 1.8% [2026-10-20 01:42:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.5862 [2026-10-20 01:42:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=9.0190 [2026-10-20 01:42:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.3700 [2026-10-20 01:42:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.6105 [2026-10-20 01:42:15][INFO][training.pretrain_engine] [Ph2|Step 176] loss=7.7401 | ppl=2298.66 | aux=0.02519 | lr=1.34e-04 | gnorm=1.078 | tok/s=29,264 | MFU=7.0% | step=143325ms | tokens=0.192B | mem=4.2/32.7GB | cost=$92.43/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 22:15 (140.6h) | Phase3: SKIPPED | SFT: Oct 26 03:00 (145.3h) | DPO (Final): Oct 26 03:34 (145.9h) [2026-10-20 01:42:15][INFO][training.pretrain_engine] Starting global step 177 (accumulating 32 micro-batches)... [2026-10-20 01:42:15][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 01:42:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.0508 [2026-10-20 01:42:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.5034 [2026-10-20 01:42:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.9574 [2026-10-20 01:42:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.3298 [2026-10-20 01:42:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.1134 [2026-10-20 01:42:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.2260 [2026-10-20 01:42:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.0169 [2026-10-20 01:42:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.2233 [2026-10-20 01:43:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.1803 [2026-10-20 01:43:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.1549 [2026-10-20 01:43:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.3313 [2026-10-20 01:43:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.9787 [2026-10-20 01:43:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4322 [2026-10-20 01:43:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4255 [2026-10-20 01:43:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.3043 [2026-10-20 01:43:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.6443 [2026-10-20 01:43:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.6544 [2026-10-20 01:43:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.5812 [2026-10-20 01:43:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.4537 [2026-10-20 01:43:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2321 [2026-10-20 01:44:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.8385 [2026-10-20 01:44:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.4743 [2026-10-20 01:44:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.5432 [2026-10-20 01:44:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.7467 [2026-10-20 01:44:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.3175 [2026-10-20 01:44:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.3659 [2026-10-20 01:44:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.1522 [2026-10-20 01:44:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.8355 [2026-10-20 01:44:34][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,556 | MFU: 1.8% [2026-10-20 01:44:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.4325 [2026-10-20 01:44:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.5660 [2026-10-20 01:44:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.6013 [2026-10-20 01:44:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.5117 [2026-10-20 01:44:39][INFO][training.pretrain_engine] [Ph2|Step 177] loss=7.6163 | ppl=2031.10 | aux=0.02515 | lr=1.34e-04 | gnorm=1.430 | tok/s=29,264 | MFU=7.0% | step=143325ms | tokens=0.193B | mem=4.2/32.7GB | cost=$92.96/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 22:17 (140.5h) | Phase3: SKIPPED | SFT: Oct 26 03:02 (145.3h) | DPO (Final): Oct 26 03:36 (145.9h) [2026-10-20 01:44:39][INFO][training.pretrain_engine] Starting global step 178 (accumulating 32 micro-batches)... [2026-10-20 01:44:39][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 01:44:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2353 [2026-10-20 01:44:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.6990 [2026-10-20 01:44:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.8551 [2026-10-20 01:44:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.4826 [2026-10-20 01:45:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.8204 [2026-10-20 01:45:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4010 [2026-10-20 01:45:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4405 [2026-10-20 01:45:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.9721 [2026-10-20 01:45:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4853 [2026-10-20 01:45:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3543 [2026-10-20 01:45:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.2470 [2026-10-20 01:45:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1253 [2026-10-20 01:45:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1832 [2026-10-20 01:45:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.0313 [2026-10-20 01:45:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.6820 [2026-10-20 01:45:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4665 [2026-10-20 01:46:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.5765 [2026-10-20 01:46:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3266 [2026-10-20 01:46:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2375 [2026-10-20 01:46:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3688 [2026-10-20 01:46:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.5730 [2026-10-20 01:46:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.2040 [2026-10-20 01:46:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.6345 [2026-10-20 01:46:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.6562 [2026-10-20 01:46:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.4763 [2026-10-20 01:46:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.5783 [2026-10-20 01:46:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3309 [2026-10-20 01:46:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.4157 [2026-10-20 01:46:57][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,575 | MFU: 1.8% [2026-10-20 01:47:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2679 [2026-10-20 01:47:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.5228 [2026-10-20 01:47:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.6319 [2026-10-20 01:47:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4653 [2026-10-20 01:47:02][INFO][training.pretrain_engine] [Ph2|Step 178] loss=7.6388 | ppl=2077.25 | aux=0.02518 | lr=1.35e-04 | gnorm=1.188 | tok/s=29,265 | MFU=7.0% | step=143320ms | tokens=0.194B | mem=4.2/32.7GB | cost=$93.48/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 22:19 (140.5h) | Phase3: SKIPPED | SFT: Oct 26 03:03 (145.3h) | DPO (Final): Oct 26 03:38 (145.8h) [2026-10-20 01:47:02][INFO][training.pretrain_engine] Starting global step 179 (accumulating 32 micro-batches)... [2026-10-20 01:47:02][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 01:47:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.3052 [2026-10-20 01:47:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.6786 [2026-10-20 01:47:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.4973 [2026-10-20 01:47:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.8053 [2026-10-20 01:47:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.3765 [2026-10-20 01:47:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4297 [2026-10-20 01:47:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.7499 [2026-10-20 01:47:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9563 [2026-10-20 01:47:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.6315 [2026-10-20 01:47:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3284 [2026-10-20 01:47:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.3154 [2026-10-20 01:47:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.6743 [2026-10-20 01:48:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.8719 [2026-10-20 01:48:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.6102 [2026-10-20 01:48:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.5029 [2026-10-20 01:48:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.6969 [2026-10-20 01:48:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3766 [2026-10-20 01:48:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.5750 [2026-10-20 01:48:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2782 [2026-10-20 01:48:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.8161 [2026-10-20 01:48:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.5474 [2026-10-20 01:48:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.6965 [2026-10-20 01:48:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1021 [2026-10-20 01:48:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2985 [2026-10-20 01:49:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.5606 [2026-10-20 01:49:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.4223 [2026-10-20 01:49:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.0973 [2026-10-20 01:49:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.1249 [2026-10-20 01:49:20][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,604 | MFU: 1.8% [2026-10-20 01:49:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.0268 [2026-10-20 01:49:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.7800 [2026-10-20 01:49:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4826 [2026-10-20 01:49:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.6425 [2026-10-20 01:49:25][INFO][training.pretrain_engine] [Ph2|Step 179] loss=7.7018 | ppl=2212.31 | aux=0.02521 | lr=1.35e-04 | gnorm=1.570 | tok/s=29,267 | MFU=7.0% | step=143313ms | tokens=0.195B | mem=4.2/32.7GB | cost=$94.00/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 22:20 (140.5h) | Phase3: SKIPPED | SFT: Oct 26 03:05 (145.3h) | DPO (Final): Oct 26 03:39 (145.8h) [2026-10-20 01:49:25][INFO][training.pretrain_engine] Starting global step 180 (accumulating 32 micro-batches)... [2026-10-20 01:49:25][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 01:49:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.4073 [2026-10-20 01:49:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.5029 [2026-10-20 01:49:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.7086 [2026-10-20 01:49:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.8415 [2026-10-20 01:50:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.3010 [2026-10-20 01:50:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.6769 [2026-10-20 01:50:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8775 [2026-10-20 01:50:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.3267 [2026-10-20 01:50:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.6222 [2026-10-20 01:50:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.5148 [2026-10-20 01:50:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4790 [2026-10-20 01:50:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.2704 [2026-10-20 01:50:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.1651 [2026-10-20 01:50:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.7334 [2026-10-20 01:50:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.9272 [2026-10-20 01:50:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.4125 [2026-10-20 01:50:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.8671 [2026-10-20 01:50:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.4029 [2026-10-20 01:50:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.8791 [2026-10-20 01:50:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.6380 [2026-10-20 01:51:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.9641 [2026-10-20 01:51:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.3323 [2026-10-20 01:51:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.5860 [2026-10-20 01:51:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.4642 [2026-10-20 01:51:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.3695 [2026-10-20 01:51:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.7088 [2026-10-20 01:51:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.2784 [2026-10-20 01:51:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3268 [2026-10-20 01:51:43][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,618 | MFU: 1.8% [2026-10-20 01:51:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.5695 [2026-10-20 01:51:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4512 [2026-10-20 01:51:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.6684 [2026-10-20 01:51:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4803 [2026-10-20 01:51:48][INFO][training.pretrain_engine] [Ph2|Step 180] loss=7.6641 | ppl=2130.48 | aux=0.02519 | lr=1.36e-04 | gnorm=1.266 | tok/s=29,268 | MFU=7.0% | step=143306ms | tokens=0.196B | mem=4.2/32.7GB | cost=$94.53/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 22:21 (140.5h) | Phase3: SKIPPED | SFT: Oct 26 03:06 (145.2h) | DPO (Final): Oct 26 03:40 (145.8h) [2026-10-20 01:51:51][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step180 | Tokens: 0.196B | Spent: $94.54 | Remaining: $305.46 [2026-10-20 01:51:51][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step180 [2026-10-20 01:51:51][INFO][training.pretrain_engine] Starting global step 181 (accumulating 32 micro-batches)... [2026-10-20 01:51:51][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 01:51:51][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 01:51:51][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 01:51:51][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 01:51:51][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step180 to Phase-Technologies/forge-500m-pretrain (path: phase2_step180)... [2026-10-20 01:51:52][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 01:52:00][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 01:52:00][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step180 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 01:52:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.6887 [2026-10-20 01:52:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.6994 [2026-10-20 01:52:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2293 [2026-10-20 01:52:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.6290 [2026-10-20 01:52:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.8236 [2026-10-20 01:52:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4305 [2026-10-20 01:52:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.8077 [2026-10-20 01:52:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.6705 [2026-10-20 01:52:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.9259 [2026-10-20 01:52:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.6821 [2026-10-20 01:52:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.7384 [2026-10-20 01:52:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.6927 [2026-10-20 01:52:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.5351 [2026-10-20 01:52:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.2422 [2026-10-20 01:53:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.9542 [2026-10-20 01:53:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.8142 [2026-10-20 01:53:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.9933 [2026-10-20 01:53:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.8967 [2026-10-20 01:53:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.6667 [2026-10-20 01:53:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.6685 [2026-10-20 01:53:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.9530 [2026-10-20 01:53:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.8605 [2026-10-20 01:53:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.8560 [2026-10-20 01:53:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.7301 [2026-10-20 01:53:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.4152 [2026-10-20 01:53:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2274 [2026-10-20 01:53:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2109 [2026-10-20 01:53:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.5563 [2026-10-20 01:54:09][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,585 | MFU: 1.8% [2026-10-20 01:54:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.6511 [2026-10-20 01:54:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.6094 [2026-10-20 01:54:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4498 [2026-10-20 01:54:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.0789 [2026-10-20 01:54:14][INFO][training.pretrain_engine] [Ph2|Step 181] loss=7.6639 | ppl=2130.02 | aux=0.02517 | lr=1.36e-04 | gnorm=1.094 | tok/s=29,270 | MFU=7.0% | step=143295ms | tokens=0.197B | mem=4.2/32.7GB | cost=$95.06/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 22:23 (140.5h) | Phase3: SKIPPED | SFT: Oct 26 03:07 (145.2h) | DPO (Final): Oct 26 03:41 (145.8h) [2026-10-20 01:54:14][INFO][training.pretrain_engine] Starting global step 182 (accumulating 32 micro-batches)... [2026-10-20 01:54:14][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 01:54:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.5362 [2026-10-20 01:54:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.9442 [2026-10-20 01:54:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1928 [2026-10-20 01:54:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.7044 [2026-10-20 01:54:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.3612 [2026-10-20 01:54:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.6225 [2026-10-20 01:54:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1052 [2026-10-20 01:54:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.6282 [2026-10-20 01:55:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.5595 [2026-10-20 01:55:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.2068 [2026-10-20 01:55:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.2960 [2026-10-20 01:55:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.9226 [2026-10-20 01:55:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.5202 [2026-10-20 01:55:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.8052 [2026-10-20 01:55:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.5736 [2026-10-20 01:55:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.9820 [2026-10-20 01:55:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.4086 [2026-10-20 01:55:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.4061 [2026-10-20 01:55:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.7422 [2026-10-20 01:55:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2547 [2026-10-20 01:56:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.3667 [2026-10-20 01:56:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2754 [2026-10-20 01:56:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1889 [2026-10-20 01:56:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.5810 [2026-10-20 01:56:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.7140 [2026-10-20 01:56:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2762 [2026-10-20 01:56:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.5504 [2026-10-20 01:56:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.4000 [2026-10-20 01:56:31][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,617 | MFU: 1.8% [2026-10-20 01:56:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2329 [2026-10-20 01:56:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4976 [2026-10-20 01:56:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.7782 [2026-10-20 01:56:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2990 [2026-10-20 01:56:37][INFO][training.pretrain_engine] [Ph2|Step 182] loss=7.6940 | ppl=2195.16 | aux=0.02520 | lr=1.37e-04 | gnorm=1.070 | tok/s=29,270 | MFU=7.0% | step=143296ms | tokens=0.198B | mem=4.2/32.7GB | cost=$95.58/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 22:24 (140.5h) | Phase3: SKIPPED | SFT: Oct 26 03:09 (145.2h) | DPO (Final): Oct 26 03:43 (145.8h) [2026-10-20 01:56:37][INFO][training.pretrain_engine] Starting global step 183 (accumulating 32 micro-batches)... [2026-10-20 01:56:37][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 01:56:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.1068 [2026-10-20 01:56:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2042 [2026-10-20 01:56:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0930 [2026-10-20 01:56:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.6113 [2026-10-20 01:57:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.5012 [2026-10-20 01:57:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2709 [2026-10-20 01:57:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4360 [2026-10-20 01:57:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0423 [2026-10-20 01:57:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.9881 [2026-10-20 01:57:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.6294 [2026-10-20 01:57:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0966 [2026-10-20 01:57:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3464 [2026-10-20 01:57:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4575 [2026-10-20 01:57:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4833 [2026-10-20 01:57:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0841 [2026-10-20 01:57:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4181 [2026-10-20 01:58:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.4576 [2026-10-20 01:58:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.4411 [2026-10-20 01:58:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.5634 [2026-10-20 01:58:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.6601 [2026-10-20 01:58:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1986 [2026-10-20 01:58:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.3459 [2026-10-20 01:58:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.7565 [2026-10-20 01:58:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.5170 [2026-10-20 01:58:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0895 [2026-10-20 01:58:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.4882 [2026-10-20 01:58:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3106 [2026-10-20 01:58:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.0592 [2026-10-20 01:58:56][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,568 | MFU: 1.8% [2026-10-20 01:59:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.8193 [2026-10-20 01:59:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.6204 [2026-10-20 01:59:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.3431 [2026-10-20 01:59:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.8917 [2026-10-20 01:59:00][INFO][training.pretrain_engine] [Ph2|Step 183] loss=7.7295 | ppl=2274.50 | aux=0.02517 | lr=1.38e-04 | gnorm=1.305 | tok/s=29,271 | MFU=7.0% | step=143294ms | tokens=0.199B | mem=4.2/32.7GB | cost=$96.11/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 22:26 (140.5h) | Phase3: SKIPPED | SFT: Oct 26 03:11 (145.2h) | DPO (Final): Oct 26 03:45 (145.8h) [2026-10-20 01:59:00][INFO][training.pretrain_engine] Starting global step 184 (accumulating 32 micro-batches)... [2026-10-20 01:59:00][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 01:59:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.0562 [2026-10-20 01:59:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.9660 [2026-10-20 01:59:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.9907 [2026-10-20 01:59:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.8760 [2026-10-20 01:59:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.1911 [2026-10-20 01:59:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2911 [2026-10-20 01:59:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.6747 [2026-10-20 01:59:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.6043 [2026-10-20 01:59:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.8207 [2026-10-20 01:59:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.6478 [2026-10-20 01:59:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.8835 [2026-10-20 01:59:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4865 [2026-10-20 02:00:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2747 [2026-10-20 02:00:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.8588 [2026-10-20 02:00:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2075 [2026-10-20 02:00:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2329 [2026-10-20 02:00:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.5402 [2026-10-20 02:00:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.4971 [2026-10-20 02:00:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.4365 [2026-10-20 02:00:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3359 [2026-10-20 02:00:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2689 [2026-10-20 02:00:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.8755 [2026-10-20 02:00:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.2605 [2026-10-20 02:00:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.9136 [2026-10-20 02:01:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1956 [2026-10-20 02:01:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3168 [2026-10-20 02:01:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.7074 [2026-10-20 02:01:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3920 [2026-10-20 02:01:19][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,578 | MFU: 1.8% [2026-10-20 02:01:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4838 [2026-10-20 02:01:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4112 [2026-10-20 02:01:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.8166 [2026-10-20 02:01:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.1341 [2026-10-20 02:01:23][INFO][training.pretrain_engine] [Ph2|Step 184] loss=7.5679 | ppl=1935.00 | aux=0.02514 | lr=1.38e-04 | gnorm=1.164 | tok/s=29,271 | MFU=7.0% | step=143292ms | tokens=0.200B | mem=4.2/32.7GB | cost=$96.63/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 22:28 (140.4h) | Phase3: SKIPPED | SFT: Oct 26 03:12 (145.2h) | DPO (Final): Oct 26 03:47 (145.8h) [2026-10-20 02:01:23][INFO][training.pretrain_engine] Starting global step 185 (accumulating 32 micro-batches)... [2026-10-20 02:01:23][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 02:01:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2152 [2026-10-20 02:01:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2114 [2026-10-20 02:01:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.5538 [2026-10-20 02:01:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2508 [2026-10-20 02:01:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4929 [2026-10-20 02:01:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.7945 [2026-10-20 02:01:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.5930 [2026-10-20 02:01:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.8596 [2026-10-20 02:02:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.7903 [2026-10-20 02:02:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.7116 [2026-10-20 02:02:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3657 [2026-10-20 02:02:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.2414 [2026-10-20 02:02:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.5193 [2026-10-20 02:02:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4083 [2026-10-20 02:02:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4442 [2026-10-20 02:02:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1175 [2026-10-20 02:02:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.2247 [2026-10-20 02:02:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.1210 [2026-10-20 02:02:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.0292 [2026-10-20 02:02:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.0869 [2026-10-20 02:03:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0863 [2026-10-20 02:03:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.5846 [2026-10-20 02:03:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.5136 [2026-10-20 02:03:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.3057 [2026-10-20 02:03:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3681 [2026-10-20 02:03:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.6332 [2026-10-20 02:03:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.4690 [2026-10-20 02:03:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.6183 [2026-10-20 02:03:42][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,569 | MFU: 1.8% [2026-10-20 02:03:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.6663 [2026-10-20 02:03:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.9130 [2026-10-20 02:03:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.6041 [2026-10-20 02:03:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4522 [2026-10-20 02:03:46][INFO][training.pretrain_engine] [Ph2|Step 185] loss=7.5951 | ppl=1988.39 | aux=0.02517 | lr=1.39e-04 | gnorm=0.953 | tok/s=29,272 | MFU=7.0% | step=143287ms | tokens=0.201B | mem=4.2/32.7GB | cost=$97.15/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 22:29 (140.4h) | Phase3: SKIPPED | SFT: Oct 26 03:14 (145.2h) | DPO (Final): Oct 26 03:48 (145.7h) [2026-10-20 02:03:46][INFO][training.pretrain_engine] Starting global step 186 (accumulating 32 micro-batches)... [2026-10-20 02:03:46][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 02:04:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.7876 [2026-10-20 02:04:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.8271 [2026-10-20 02:04:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.7680 [2026-10-20 02:04:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.9736 [2026-10-20 02:04:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4168 [2026-10-20 02:04:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4494 [2026-10-20 02:04:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.5424 [2026-10-20 02:04:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.3416 [2026-10-20 02:04:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.2878 [2026-10-20 02:04:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3796 [2026-10-20 02:04:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.7002 [2026-10-20 02:04:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.6261 [2026-10-20 02:04:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4371 [2026-10-20 02:04:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2611 [2026-10-20 02:04:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.3201 [2026-10-20 02:04:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.0333 [2026-10-20 02:05:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.6473 [2026-10-20 02:05:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.7227 [2026-10-20 02:05:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2556 [2026-10-20 02:05:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.7369 [2026-10-20 02:05:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.6259 [2026-10-20 02:05:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.0207 [2026-10-20 02:05:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.5007 [2026-10-20 02:05:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.8308 [2026-10-20 02:05:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.5004 [2026-10-20 02:05:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1389 [2026-10-20 02:05:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5689 [2026-10-20 02:05:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.5514 [2026-10-20 02:06:05][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,557 | MFU: 1.8% [2026-10-20 02:06:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.5983 [2026-10-20 02:06:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.8282 [2026-10-20 02:06:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.8286 [2026-10-20 02:06:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.6469 [2026-10-20 02:06:10][INFO][training.pretrain_engine] [Ph2|Step 186] loss=7.5345 | ppl=1871.56 | aux=0.02517 | lr=1.39e-04 | gnorm=1.312 | tok/s=29,272 | MFU=7.0% | step=143287ms | tokens=0.202B | mem=4.2/32.7GB | cost=$97.68/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 22:31 (140.4h) | Phase3: SKIPPED | SFT: Oct 26 03:16 (145.2h) | DPO (Final): Oct 26 03:50 (145.7h) [2026-10-20 02:06:10][INFO][training.pretrain_engine] Starting global step 187 (accumulating 32 micro-batches)... [2026-10-20 02:06:10][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 02:06:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.7841 [2026-10-20 02:06:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.7053 [2026-10-20 02:06:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.7097 [2026-10-20 02:06:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.8300 [2026-10-20 02:06:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.3775 [2026-10-20 02:06:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1761 [2026-10-20 02:06:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.7176 [2026-10-20 02:06:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2820 [2026-10-20 02:07:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4036 [2026-10-20 02:07:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.2342 [2026-10-20 02:07:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.2742 [2026-10-20 02:07:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.8091 [2026-10-20 02:07:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.6136 [2026-10-20 02:07:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.8064 [2026-10-20 02:07:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2660 [2026-10-20 02:07:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4403 [2026-10-20 02:07:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.4262 [2026-10-20 02:07:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.9651 [2026-10-20 02:07:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2726 [2026-10-20 02:07:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3667 [2026-10-20 02:07:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.5936 [2026-10-20 02:07:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1408 [2026-10-20 02:07:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.4179 [2026-10-20 02:07:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.7223 [2026-10-20 02:08:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.7112 [2026-10-20 02:08:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3638 [2026-10-20 02:08:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2985 [2026-10-20 02:08:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.6523 [2026-10-20 02:08:28][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,589 | MFU: 1.8% [2026-10-20 02:08:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3232 [2026-10-20 02:08:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.9274 [2026-10-20 02:08:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.9682 [2026-10-20 02:08:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7760 [2026-10-20 02:08:32][INFO][training.pretrain_engine] [Ph2|Step 187] loss=7.5532 | ppl=1906.79 | aux=0.02515 | lr=1.40e-04 | gnorm=0.641 | tok/s=29,274 | MFU=7.0% | step=143279ms | tokens=0.203B | mem=4.2/32.7GB | cost=$98.20/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 22:32 (140.4h) | Phase3: SKIPPED | SFT: Oct 26 03:17 (145.1h) | DPO (Final): Oct 26 03:51 (145.7h) [2026-10-20 02:08:32][INFO][training.pretrain_engine] Starting global step 188 (accumulating 32 micro-batches)... [2026-10-20 02:08:32][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 02:08:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.6882 [2026-10-20 02:08:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.6836 [2026-10-20 02:08:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.6908 [2026-10-20 02:08:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.7285 [2026-10-20 02:09:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.8492 [2026-10-20 02:09:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.1987 [2026-10-20 02:09:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.1771 [2026-10-20 02:09:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.5076 [2026-10-20 02:09:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.7256 [2026-10-20 02:09:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.5424 [2026-10-20 02:09:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3627 [2026-10-20 02:09:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4050 [2026-10-20 02:09:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0603 [2026-10-20 02:09:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2319 [2026-10-20 02:09:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2643 [2026-10-20 02:09:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4690 [2026-10-20 02:10:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.6325 [2026-10-20 02:10:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.4211 [2026-10-20 02:10:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.6411 [2026-10-20 02:10:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.4817 [2026-10-20 02:10:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.3750 [2026-10-20 02:10:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.5507 [2026-10-20 02:10:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.7082 [2026-10-20 02:10:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.5257 [2026-10-20 02:10:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.8148 [2026-10-20 02:10:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.7484 [2026-10-20 02:10:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.2337 [2026-10-20 02:10:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.3931 [2026-10-20 02:10:51][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,594 | MFU: 1.8% [2026-10-20 02:10:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.9425 [2026-10-20 02:10:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4748 [2026-10-20 02:10:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.7559 [2026-10-20 02:10:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7258 [2026-10-20 02:10:56][INFO][training.pretrain_engine] [Ph2|Step 188] loss=7.5559 | ppl=1911.90 | aux=0.02519 | lr=1.40e-04 | gnorm=1.062 | tok/s=29,274 | MFU=7.0% | step=143278ms | tokens=0.204B | mem=4.2/32.7GB | cost=$98.72/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 22:34 (140.4h) | Phase3: SKIPPED | SFT: Oct 26 03:19 (145.1h) | DPO (Final): Oct 26 03:53 (145.7h) [2026-10-20 02:10:56][INFO][training.pretrain_engine] Starting global step 189 (accumulating 32 micro-batches)... [2026-10-20 02:10:56][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 02:11:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.5438 [2026-10-20 02:11:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2890 [2026-10-20 02:11:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.4066 [2026-10-20 02:11:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2894 [2026-10-20 02:11:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4823 [2026-10-20 02:11:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.5046 [2026-10-20 02:11:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.5582 [2026-10-20 02:11:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4284 [2026-10-20 02:11:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3645 [2026-10-20 02:11:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4326 [2026-10-20 02:11:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.6040 [2026-10-20 02:11:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.6855 [2026-10-20 02:12:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0176 [2026-10-20 02:12:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9404 [2026-10-20 02:12:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.3486 [2026-10-20 02:12:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8516 [2026-10-20 02:12:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.4295 [2026-10-20 02:12:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.4166 [2026-10-20 02:12:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2610 [2026-10-20 02:12:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2403 [2026-10-20 02:12:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2972 [2026-10-20 02:12:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.6985 [2026-10-20 02:12:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.5857 [2026-10-20 02:12:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.3978 [2026-10-20 02:13:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.7395 [2026-10-20 02:13:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.4314 [2026-10-20 02:13:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2696 [2026-10-20 02:13:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.6359 [2026-10-20 02:13:15][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,538 | MFU: 1.8% [2026-10-20 02:13:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.6142 [2026-10-20 02:13:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.6950 [2026-10-20 02:13:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.7774 [2026-10-20 02:13:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.7442 [2026-10-20 02:13:19][INFO][training.pretrain_engine] [Ph2|Step 189] loss=7.5820 | ppl=1962.61 | aux=0.02517 | lr=1.41e-04 | gnorm=1.125 | tok/s=29,273 | MFU=7.0% | step=143284ms | tokens=0.206B | mem=4.2/32.7GB | cost=$99.25/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 22:36 (140.4h) | Phase3: SKIPPED | SFT: Oct 26 03:21 (145.1h) | DPO (Final): Oct 26 03:55 (145.7h) [2026-10-20 02:13:23][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step189 | Tokens: 0.206B | Spent: $99.26 | Remaining: $300.74 [2026-10-20 02:13:23][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step189 [2026-10-20 02:13:23][INFO][training.pretrain_engine] Starting global step 190 (accumulating 32 micro-batches)... [2026-10-20 02:13:23][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 02:13:23][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 02:13:23][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 02:13:23][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 02:13:23][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step189 to Phase-Technologies/forge-500m-pretrain (path: phase2_step189)... [2026-10-20 02:13:23][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 02:13:33][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 02:13:33][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step189 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 02:13:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.3869 [2026-10-20 02:13:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.8021 [2026-10-20 02:13:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.5614 [2026-10-20 02:13:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.3879 [2026-10-20 02:13:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.6499 [2026-10-20 02:13:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.9511 [2026-10-20 02:13:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.3596 [2026-10-20 02:13:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.7683 [2026-10-20 02:14:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.0426 [2026-10-20 02:14:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.7283 [2026-10-20 02:14:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4718 [2026-10-20 02:14:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4062 [2026-10-20 02:14:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.0381 [2026-10-20 02:14:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.9557 [2026-10-20 02:14:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.9648 [2026-10-20 02:14:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.8388 [2026-10-20 02:14:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3744 [2026-10-20 02:14:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.7040 [2026-10-20 02:14:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3679 [2026-10-20 02:14:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2912 [2026-10-20 02:15:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1479 [2026-10-20 02:15:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2860 [2026-10-20 02:15:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.6425 [2026-10-20 02:15:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2895 [2026-10-20 02:15:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.5781 [2026-10-20 02:15:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3755 [2026-10-20 02:15:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.6917 [2026-10-20 02:15:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.6308 [2026-10-20 02:15:42][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,527 | MFU: 1.8% [2026-10-20 02:15:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.9676 [2026-10-20 02:15:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2273 [2026-10-20 02:15:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3568 [2026-10-20 02:15:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3057 [2026-10-20 02:15:47][INFO][training.pretrain_engine] [Ph2|Step 190] loss=7.5752 | ppl=1949.30 | aux=0.02516 | lr=1.42e-04 | gnorm=1.148 | tok/s=29,271 | MFU=7.0% | step=143291ms | tokens=0.207B | mem=4.2/32.7GB | cost=$99.79/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 22:39 (140.4h) | Phase3: SKIPPED | SFT: Oct 26 03:23 (145.1h) | DPO (Final): Oct 26 03:57 (145.7h) [2026-10-20 02:15:47][INFO][training.pretrain_engine] Starting global step 191 (accumulating 32 micro-batches)... [2026-10-20 02:15:47][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 02:16:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1092 [2026-10-20 02:16:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.4841 [2026-10-20 02:16:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.3643 [2026-10-20 02:16:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2337 [2026-10-20 02:16:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.8644 [2026-10-20 02:16:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.0166 [2026-10-20 02:16:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.9073 [2026-10-20 02:16:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.7692 [2026-10-20 02:16:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3047 [2026-10-20 02:16:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.2224 [2026-10-20 02:16:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3599 [2026-10-20 02:16:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1754 [2026-10-20 02:16:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.6688 [2026-10-20 02:16:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2086 [2026-10-20 02:16:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4000 [2026-10-20 02:16:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.3458 [2026-10-20 02:17:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1435 [2026-10-20 02:17:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1454 [2026-10-20 02:17:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0293 [2026-10-20 02:17:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0720 [2026-10-20 02:17:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.4545 [2026-10-20 02:17:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.5690 [2026-10-20 02:17:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1785 [2026-10-20 02:17:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1145 [2026-10-20 02:17:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1633 [2026-10-20 02:17:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.4008 [2026-10-20 02:17:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.5773 [2026-10-20 02:17:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.8622 [2026-10-20 02:18:06][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,548 | MFU: 1.8% [2026-10-20 02:18:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4022 [2026-10-20 02:18:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.5989 [2026-10-20 02:18:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.1765 [2026-10-20 02:18:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2383 [2026-10-20 02:18:10][INFO][training.pretrain_engine] [Ph2|Step 191] loss=7.4649 | ppl=1745.71 | aux=0.02517 | lr=1.42e-04 | gnorm=0.879 | tok/s=29,269 | MFU=7.0% | step=143300ms | tokens=0.208B | mem=4.2/32.7GB | cost=$100.31/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 22:41 (140.4h) | Phase3: SKIPPED | SFT: Oct 26 03:25 (145.1h) | DPO (Final): Oct 26 04:00 (145.7h) [2026-10-20 02:18:10][INFO][training.pretrain_engine] Starting global step 192 (accumulating 32 micro-batches)... [2026-10-20 02:18:10][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 02:18:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.4587 [2026-10-20 02:18:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.6914 [2026-10-20 02:18:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1535 [2026-10-20 02:18:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1328 [2026-10-20 02:18:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4374 [2026-10-20 02:18:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.5847 [2026-10-20 02:18:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.8820 [2026-10-20 02:18:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4844 [2026-10-20 02:19:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.5451 [2026-10-20 02:19:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.5796 [2026-10-20 02:19:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4199 [2026-10-20 02:19:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3868 [2026-10-20 02:19:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.7150 [2026-10-20 02:19:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.9286 [2026-10-20 02:19:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.0052 [2026-10-20 02:19:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4542 [2026-10-20 02:19:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2282 [2026-10-20 02:19:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.4460 [2026-10-20 02:19:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3153 [2026-10-20 02:19:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.4732 [2026-10-20 02:19:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.5770 [2026-10-20 02:19:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.7107 [2026-10-20 02:19:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.5632 [2026-10-20 02:19:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.6534 [2026-10-20 02:20:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9867 [2026-10-20 02:20:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8493 [2026-10-20 02:20:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1180 [2026-10-20 02:20:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2750 [2026-10-20 02:20:29][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,554 | MFU: 1.8% [2026-10-20 02:20:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.5844 [2026-10-20 02:20:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.1784 [2026-10-20 02:20:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3333 [2026-10-20 02:20:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0325 [2026-10-20 02:20:34][INFO][training.pretrain_engine] [Ph2|Step 192] loss=7.4665 | ppl=1748.45 | aux=0.02513 | lr=1.43e-04 | gnorm=0.863 | tok/s=29,269 | MFU=7.0% | step=143300ms | tokens=0.209B | mem=4.2/32.7GB | cost=$100.83/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 22:43 (140.4h) | Phase3: SKIPPED | SFT: Oct 26 03:27 (145.1h) | DPO (Final): Oct 26 04:01 (145.7h) [2026-10-20 02:20:34][INFO][training.pretrain_engine] Starting global step 193 (accumulating 32 micro-batches)... [2026-10-20 02:20:34][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 02:20:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.4087 [2026-10-20 02:20:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.5217 [2026-10-20 02:20:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.9738 [2026-10-20 02:20:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.6774 [2026-10-20 02:21:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.9559 [2026-10-20 02:21:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.1054 [2026-10-20 02:21:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.5251 [2026-10-20 02:21:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.5326 [2026-10-20 02:21:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.0992 [2026-10-20 02:21:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.9586 [2026-10-20 02:21:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1028 [2026-10-20 02:21:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3136 [2026-10-20 02:21:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2579 [2026-10-20 02:21:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2189 [2026-10-20 02:21:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.6969 [2026-10-20 02:21:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.9550 [2026-10-20 02:22:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0950 [2026-10-20 02:22:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1459 [2026-10-20 02:22:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2375 [2026-10-20 02:22:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.5676 [2026-10-20 02:22:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.7170 [2026-10-20 02:22:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1908 [2026-10-20 02:22:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.3881 [2026-10-20 02:22:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.3154 [2026-10-20 02:22:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.4984 [2026-10-20 02:22:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3823 [2026-10-20 02:22:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3818 [2026-10-20 02:22:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2194 [2026-10-20 02:22:52][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,566 | MFU: 1.8% [2026-10-20 02:22:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3270 [2026-10-20 02:22:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.1230 [2026-10-20 02:22:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9944 [2026-10-20 02:22:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0907 [2026-10-20 02:22:57][INFO][training.pretrain_engine] [Ph2|Step 193] loss=7.3882 | ppl=1616.78 | aux=0.02517 | lr=1.43e-04 | gnorm=1.500 | tok/s=29,270 | MFU=7.0% | step=143297ms | tokens=0.210B | mem=4.2/32.7GB | cost=$101.36/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 22:44 (140.4h) | Phase3: SKIPPED | SFT: Oct 26 03:29 (145.1h) | DPO (Final): Oct 26 04:03 (145.7h) [2026-10-20 02:22:57][INFO][training.pretrain_engine] Starting global step 194 (accumulating 32 micro-batches)... [2026-10-20 02:22:57][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 02:23:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.5294 [2026-10-20 02:23:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.5530 [2026-10-20 02:23:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.7916 [2026-10-20 02:23:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.4141 [2026-10-20 02:23:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.7428 [2026-10-20 02:23:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.8056 [2026-10-20 02:23:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.7751 [2026-10-20 02:23:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.5649 [2026-10-20 02:23:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.2087 [2026-10-20 02:23:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0252 [2026-10-20 02:23:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3860 [2026-10-20 02:23:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.8752 [2026-10-20 02:24:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.5350 [2026-10-20 02:24:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.5393 [2026-10-20 02:24:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.3173 [2026-10-20 02:24:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.3999 [2026-10-20 02:24:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.5788 [2026-10-20 02:24:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.6789 [2026-10-20 02:24:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.7614 [2026-10-20 02:24:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.5313 [2026-10-20 02:24:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.9853 [2026-10-20 02:24:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.1787 [2026-10-20 02:24:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.1618 [2026-10-20 02:24:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9499 [2026-10-20 02:25:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1146 [2026-10-20 02:25:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9348 [2026-10-20 02:25:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6372 [2026-10-20 02:25:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3491 [2026-10-20 02:25:15][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,573 | MFU: 1.8% [2026-10-20 02:25:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.8643 [2026-10-20 02:25:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.5177 [2026-10-20 02:25:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.5269 [2026-10-20 02:25:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.6509 [2026-10-20 02:25:20][INFO][training.pretrain_engine] [Ph2|Step 194] loss=7.4845 | ppl=1780.20 | aux=0.02517 | lr=1.44e-04 | gnorm=0.840 | tok/s=29,270 | MFU=7.0% | step=143296ms | tokens=0.211B | mem=4.2/32.7GB | cost=$101.88/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 22:46 (140.4h) | Phase3: SKIPPED | SFT: Oct 26 03:31 (145.1h) | DPO (Final): Oct 26 04:05 (145.7h) [2026-10-20 02:25:20][INFO][training.pretrain_engine] Starting global step 195 (accumulating 32 micro-batches)... [2026-10-20 02:25:20][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 02:25:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2171 [2026-10-20 02:25:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.3150 [2026-10-20 02:25:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.5317 [2026-10-20 02:25:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.7941 [2026-10-20 02:25:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.3426 [2026-10-20 02:25:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4558 [2026-10-20 02:25:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.8831 [2026-10-20 02:25:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.7085 [2026-10-20 02:26:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.7588 [2026-10-20 02:26:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4552 [2026-10-20 02:26:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.6569 [2026-10-20 02:26:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3737 [2026-10-20 02:26:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.1571 [2026-10-20 02:26:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.9279 [2026-10-20 02:26:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.7745 [2026-10-20 02:26:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.5653 [2026-10-20 02:26:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0937 [2026-10-20 02:26:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2985 [2026-10-20 02:26:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0131 [2026-10-20 02:26:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3186 [2026-10-20 02:27:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1954 [2026-10-20 02:27:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0301 [2026-10-20 02:27:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1839 [2026-10-20 02:27:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1353 [2026-10-20 02:27:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.4382 [2026-10-20 02:27:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.4346 [2026-10-20 02:27:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3006 [2026-10-20 02:27:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.5624 [2026-10-20 02:27:38][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,585 | MFU: 1.8% [2026-10-20 02:27:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.6050 [2026-10-20 02:27:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.7645 [2026-10-20 02:27:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9764 [2026-10-20 02:27:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4120 [2026-10-20 02:27:43][INFO][training.pretrain_engine] [Ph2|Step 195] loss=7.5129 | ppl=1831.57 | aux=0.02517 | lr=1.44e-04 | gnorm=1.398 | tok/s=29,271 | MFU=7.0% | step=143292ms | tokens=0.212B | mem=4.2/32.7GB | cost=$102.40/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 22:47 (140.3h) | Phase3: SKIPPED | SFT: Oct 26 03:32 (145.1h) | DPO (Final): Oct 26 04:06 (145.7h) [2026-10-20 02:27:43][INFO][training.pretrain_engine] Starting global step 196 (accumulating 32 micro-batches)... [2026-10-20 02:27:43][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 02:28:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.0179 [2026-10-20 02:28:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.6876 [2026-10-20 02:28:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.5645 [2026-10-20 02:28:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.8337 [2026-10-20 02:28:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1127 [2026-10-20 02:28:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.3842 [2026-10-20 02:28:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2966 [2026-10-20 02:28:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2007 [2026-10-20 02:28:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.5724 [2026-10-20 02:28:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3325 [2026-10-20 02:28:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.5132 [2026-10-20 02:28:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4352 [2026-10-20 02:28:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.5787 [2026-10-20 02:28:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.6263 [2026-10-20 02:28:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.3107 [2026-10-20 02:28:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.3074 [2026-10-20 02:29:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9642 [2026-10-20 02:29:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8973 [2026-10-20 02:29:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0946 [2026-10-20 02:29:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.6792 [2026-10-20 02:29:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0556 [2026-10-20 02:29:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.4179 [2026-10-20 02:29:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.5379 [2026-10-20 02:29:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1256 [2026-10-20 02:29:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.8160 [2026-10-20 02:29:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3497 [2026-10-20 02:29:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.9641 [2026-10-20 02:29:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.4894 [2026-10-20 02:30:01][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,576 | MFU: 1.8% [2026-10-20 02:30:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.7376 [2026-10-20 02:30:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.5038 [2026-10-20 02:30:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.8127 [2026-10-20 02:30:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.0612 [2026-10-20 02:30:05][INFO][training.pretrain_engine] [Ph2|Step 196] loss=7.4893 | ppl=1788.76 | aux=0.02517 | lr=1.45e-04 | gnorm=1.000 | tok/s=29,272 | MFU=7.0% | step=143288ms | tokens=0.213B | mem=4.2/32.7GB | cost=$102.93/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 22:49 (140.3h) | Phase3: SKIPPED | SFT: Oct 26 03:34 (145.1h) | DPO (Final): Oct 26 04:08 (145.6h) [2026-10-20 02:30:05][INFO][training.pretrain_engine] Starting global step 197 (accumulating 32 micro-batches)... [2026-10-20 02:30:05][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 02:30:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.6204 [2026-10-20 02:30:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.4099 [2026-10-20 02:30:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.6978 [2026-10-20 02:30:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.4148 [2026-10-20 02:30:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4574 [2026-10-20 02:30:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.5455 [2026-10-20 02:30:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2903 [2026-10-20 02:30:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2712 [2026-10-20 02:30:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.2390 [2026-10-20 02:30:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1890 [2026-10-20 02:30:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3384 [2026-10-20 02:30:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.2051 [2026-10-20 02:31:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.9262 [2026-10-20 02:31:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.8210 [2026-10-20 02:31:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.5054 [2026-10-20 02:31:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.6130 [2026-10-20 02:31:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.0996 [2026-10-20 02:31:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.8967 [2026-10-20 02:31:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.7015 [2026-10-20 02:31:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.6328 [2026-10-20 02:31:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8523 [2026-10-20 02:31:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1438 [2026-10-20 02:31:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9736 [2026-10-20 02:31:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.3101 [2026-10-20 02:32:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3057 [2026-10-20 02:32:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.6333 [2026-10-20 02:32:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.5951 [2026-10-20 02:32:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9693 [2026-10-20 02:32:24][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,568 | MFU: 1.8% [2026-10-20 02:32:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4647 [2026-10-20 02:32:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.6528 [2026-10-20 02:32:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.6249 [2026-10-20 02:32:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.9913 [2026-10-20 02:32:29][INFO][training.pretrain_engine] [Ph2|Step 197] loss=7.4349 | ppl=1694.10 | aux=0.02516 | lr=1.46e-04 | gnorm=1.180 | tok/s=29,273 | MFU=7.0% | step=143281ms | tokens=0.214B | mem=4.2/32.7GB | cost=$103.45/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 22:50 (140.3h) | Phase3: SKIPPED | SFT: Oct 26 03:35 (145.1h) | DPO (Final): Oct 26 04:09 (145.6h) [2026-10-20 02:32:29][INFO][training.pretrain_engine] Starting global step 198 (accumulating 32 micro-batches)... [2026-10-20 02:32:29][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 02:32:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.3721 [2026-10-20 02:32:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.4564 [2026-10-20 02:32:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.3447 [2026-10-20 02:32:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1953 [2026-10-20 02:33:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0143 [2026-10-20 02:33:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4270 [2026-10-20 02:33:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2757 [2026-10-20 02:33:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.3707 [2026-10-20 02:33:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1903 [2026-10-20 02:33:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.2305 [2026-10-20 02:33:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3680 [2026-10-20 02:33:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1262 [2026-10-20 02:33:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.7454 [2026-10-20 02:33:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.9076 [2026-10-20 02:33:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.7811 [2026-10-20 02:33:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.8838 [2026-10-20 02:33:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3831 [2026-10-20 02:33:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.4331 [2026-10-20 02:33:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.4423 [2026-10-20 02:33:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1876 [2026-10-20 02:34:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.2561 [2026-10-20 02:34:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.0576 [2026-10-20 02:34:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.0809 [2026-10-20 02:34:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.1089 [2026-10-20 02:34:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.9160 [2026-10-20 02:34:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.6553 [2026-10-20 02:34:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.8245 [2026-10-20 02:34:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.9477 [2026-10-20 02:34:47][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,584 | MFU: 1.8% [2026-10-20 02:34:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2682 [2026-10-20 02:34:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2863 [2026-10-20 02:34:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.1605 [2026-10-20 02:34:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4887 [2026-10-20 02:34:51][INFO][training.pretrain_engine] [Ph2|Step 198] loss=7.4489 | ppl=1718.03 | aux=0.02518 | lr=1.46e-04 | gnorm=0.738 | tok/s=29,275 | MFU=7.0% | step=143272ms | tokens=0.215B | mem=4.2/32.7GB | cost=$103.97/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 22:52 (140.3h) | Phase3: SKIPPED | SFT: Oct 26 03:36 (145.0h) | DPO (Final): Oct 26 04:11 (145.6h) [2026-10-20 02:34:54][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step198 | Tokens: 0.215B | Spent: $103.98 | Remaining: $296.02 [2026-10-20 02:34:54][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step198 [2026-10-20 02:34:54][INFO][training.pretrain_engine] Starting global step 199 (accumulating 32 micro-batches)... [2026-10-20 02:34:54][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 02:34:54][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 02:34:54][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 02:34:55][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 02:34:55][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step198 to Phase-Technologies/forge-500m-pretrain (path: phase2_step198)... [2026-10-20 02:34:55][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 02:35:04][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 02:35:04][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step198 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 02:35:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.5436 [2026-10-20 02:35:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1927 [2026-10-20 02:35:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2117 [2026-10-20 02:35:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.5162 [2026-10-20 02:35:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2225 [2026-10-20 02:35:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.6416 [2026-10-20 02:35:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4559 [2026-10-20 02:35:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8939 [2026-10-20 02:35:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1897 [2026-10-20 02:35:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1420 [2026-10-20 02:35:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.2938 [2026-10-20 02:35:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.2199 [2026-10-20 02:36:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2925 [2026-10-20 02:36:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1904 [2026-10-20 02:36:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4366 [2026-10-20 02:36:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2041 [2026-10-20 02:36:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.4004 [2026-10-20 02:36:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3147 [2026-10-20 02:36:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.4055 [2026-10-20 02:36:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3797 [2026-10-20 02:36:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1385 [2026-10-20 02:36:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.7845 [2026-10-20 02:36:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1719 [2026-10-20 02:36:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.3114 [2026-10-20 02:36:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.0288 [2026-10-20 02:36:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.5274 [2026-10-20 02:36:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.7791 [2026-10-20 02:36:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.8357 [2026-10-20 02:37:13][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,572 | MFU: 1.8% [2026-10-20 02:37:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.6432 [2026-10-20 02:37:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.8517 [2026-10-20 02:37:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.5428 [2026-10-20 02:37:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.5828 [2026-10-20 02:37:17][INFO][training.pretrain_engine] [Ph2|Step 199] loss=7.4469 | ppl=1714.59 | aux=0.02516 | lr=1.47e-04 | gnorm=1.266 | tok/s=29,276 | MFU=7.0% | step=143268ms | tokens=0.216B | mem=4.2/32.7GB | cost=$104.50/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 22:53 (140.3h) | Phase3: SKIPPED | SFT: Oct 26 03:38 (145.0h) | DPO (Final): Oct 26 04:12 (145.6h) [2026-10-20 02:37:17][INFO][training.pretrain_engine] Starting global step 200 (accumulating 32 micro-batches)... [2026-10-20 02:37:17][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 02:37:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6399 [2026-10-20 02:37:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.3411 [2026-10-20 02:37:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9491 [2026-10-20 02:37:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1843 [2026-10-20 02:37:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4994 [2026-10-20 02:37:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.3485 [2026-10-20 02:37:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0241 [2026-10-20 02:37:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.3113 [2026-10-20 02:38:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.0388 [2026-10-20 02:38:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.6223 [2026-10-20 02:38:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.9808 [2026-10-20 02:38:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.6880 [2026-10-20 02:38:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.3804 [2026-10-20 02:38:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.3343 [2026-10-20 02:38:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4834 [2026-10-20 02:38:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1238 [2026-10-20 02:38:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.4642 [2026-10-20 02:38:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.9905 [2026-10-20 02:38:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.6859 [2026-10-20 02:38:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.5753 [2026-10-20 02:39:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.3913 [2026-10-20 02:39:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.6547 [2026-10-20 02:39:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.3208 [2026-10-20 02:39:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2186 [2026-10-20 02:39:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.8859 [2026-10-20 02:39:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.6094 [2026-10-20 02:39:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.9133 [2026-10-20 02:39:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.6006 [2026-10-20 02:39:36][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,584 | MFU: 1.8% [2026-10-20 02:39:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.7862 [2026-10-20 02:39:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.0736 [2026-10-20 02:39:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.9379 [2026-10-20 02:39:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.2713 [2026-10-20 02:39:40][INFO][training.pretrain_engine] [Ph2|Step 200] loss=7.4462 | ppl=1713.29 | aux=0.02517 | lr=1.47e-04 | gnorm=0.883 | tok/s=29,277 | MFU=7.0% | step=143263ms | tokens=0.217B | mem=4.2/32.7GB | cost=$105.03/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 22:55 (140.3h) | Phase3: SKIPPED | SFT: Oct 26 03:39 (145.0h) | DPO (Final): Oct 26 04:14 (145.6h) [2026-10-20 02:39:40][INFO][training.pretrain_engine] Starting global step 201 (accumulating 32 micro-batches)... [2026-10-20 02:39:40][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 02:39:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0702 [2026-10-20 02:39:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.4063 [2026-10-20 02:39:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9114 [2026-10-20 02:39:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2782 [2026-10-20 02:40:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2494 [2026-10-20 02:40:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0716 [2026-10-20 02:40:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0378 [2026-10-20 02:40:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1941 [2026-10-20 02:40:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3895 [2026-10-20 02:40:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3098 [2026-10-20 02:40:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4671 [2026-10-20 02:40:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1747 [2026-10-20 02:40:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2448 [2026-10-20 02:40:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.7714 [2026-10-20 02:40:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1677 [2026-10-20 02:40:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.6164 [2026-10-20 02:41:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.9893 [2026-10-20 02:41:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.9961 [2026-10-20 02:41:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.5573 [2026-10-20 02:41:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3047 [2026-10-20 02:41:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.6696 [2026-10-20 02:41:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.6023 [2026-10-20 02:41:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.8433 [2026-10-20 02:41:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.7651 [2026-10-20 02:41:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3435 [2026-10-20 02:41:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1729 [2026-10-20 02:41:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.6856 [2026-10-20 02:41:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2786 [2026-10-20 02:41:58][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,607 | MFU: 1.8% [2026-10-20 02:42:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.5957 [2026-10-20 02:42:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.8284 [2026-10-20 02:42:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.3745 [2026-10-20 02:42:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.5508 [2026-10-20 02:42:03][INFO][training.pretrain_engine] [Ph2|Step 201] loss=7.3887 | ppl=1617.58 | aux=0.02516 | lr=1.48e-04 | gnorm=1.203 | tok/s=29,278 | MFU=7.0% | step=143260ms | tokens=0.218B | mem=4.2/32.7GB | cost=$105.55/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 22:56 (140.2h) | Phase3: SKIPPED | SFT: Oct 26 03:41 (145.0h) | DPO (Final): Oct 26 04:15 (145.6h) [2026-10-20 02:42:03][INFO][training.pretrain_engine] Starting global step 202 (accumulating 32 micro-batches)... [2026-10-20 02:42:03][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 02:42:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.5239 [2026-10-20 02:42:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.7154 [2026-10-20 02:42:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.1133 [2026-10-20 02:42:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.7886 [2026-10-20 02:42:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0735 [2026-10-20 02:42:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9732 [2026-10-20 02:42:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.5570 [2026-10-20 02:42:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1264 [2026-10-20 02:42:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3405 [2026-10-20 02:42:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.6862 [2026-10-20 02:42:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9092 [2026-10-20 02:42:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1536 [2026-10-20 02:43:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1338 [2026-10-20 02:43:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1246 [2026-10-20 02:43:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2930 [2026-10-20 02:43:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.6445 [2026-10-20 02:43:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.4826 [2026-10-20 02:43:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.5203 [2026-10-20 02:43:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1807 [2026-10-20 02:43:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2064 [2026-10-20 02:43:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.3938 [2026-10-20 02:43:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2989 [2026-10-20 02:43:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2505 [2026-10-20 02:43:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.3413 [2026-10-20 02:44:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3702 [2026-10-20 02:44:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3676 [2026-10-20 02:44:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3934 [2026-10-20 02:44:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1458 [2026-10-20 02:44:21][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,588 | MFU: 1.8% [2026-10-20 02:44:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.6614 [2026-10-20 02:44:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3226 [2026-10-20 02:44:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2413 [2026-10-20 02:44:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9735 [2026-10-20 02:44:26][INFO][training.pretrain_engine] [Ph2|Step 202] loss=7.4387 | ppl=1700.61 | aux=0.02515 | lr=1.48e-04 | gnorm=1.117 | tok/s=29,278 | MFU=7.0% | step=143256ms | tokens=0.219B | mem=4.2/32.7GB | cost=$106.07/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 22:58 (140.2h) | Phase3: SKIPPED | SFT: Oct 26 03:43 (145.0h) | DPO (Final): Oct 26 04:17 (145.5h) [2026-10-20 02:44:26][INFO][training.pretrain_engine] Starting global step 203 (accumulating 32 micro-batches)... [2026-10-20 02:44:26][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 02:44:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.5915 [2026-10-20 02:44:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.5809 [2026-10-20 02:44:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.6602 [2026-10-20 02:44:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.6323 [2026-10-20 02:45:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4200 [2026-10-20 02:45:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4309 [2026-10-20 02:45:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1849 [2026-10-20 02:45:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1832 [2026-10-20 02:45:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3358 [2026-10-20 02:45:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3320 [2026-10-20 02:45:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3278 [2026-10-20 02:45:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0132 [2026-10-20 02:45:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.8677 [2026-10-20 02:45:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.5880 [2026-10-20 02:45:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.6407 [2026-10-20 02:45:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4420 [2026-10-20 02:45:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2600 [2026-10-20 02:45:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8905 [2026-10-20 02:45:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1299 [2026-10-20 02:45:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2897 [2026-10-20 02:46:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8941 [2026-10-20 02:46:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2039 [2026-10-20 02:46:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.7117 [2026-10-20 02:46:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.8822 [2026-10-20 02:46:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3536 [2026-10-20 02:46:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3808 [2026-10-20 02:46:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3405 [2026-10-20 02:46:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3082 [2026-10-20 02:46:44][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,607 | MFU: 1.8% [2026-10-20 02:46:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.1684 [2026-10-20 02:46:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8330 [2026-10-20 02:46:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8937 [2026-10-20 02:46:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0584 [2026-10-20 02:46:49][INFO][training.pretrain_engine] [Ph2|Step 203] loss=7.4370 | ppl=1697.62 | aux=0.02517 | lr=1.49e-04 | gnorm=1.164 | tok/s=29,279 | MFU=7.0% | step=143255ms | tokens=0.220B | mem=4.2/32.7GB | cost=$106.59/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 23:00 (140.2h) | Phase3: SKIPPED | SFT: Oct 26 03:44 (145.0h) | DPO (Final): Oct 26 04:18 (145.5h) [2026-10-20 02:46:49][INFO][training.pretrain_engine] Starting global step 204 (accumulating 32 micro-batches)... [2026-10-20 02:46:49][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 02:47:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.5278 [2026-10-20 02:47:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1511 [2026-10-20 02:47:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.4196 [2026-10-20 02:47:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1336 [2026-10-20 02:47:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2766 [2026-10-20 02:47:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.7114 [2026-10-20 02:47:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.7348 [2026-10-20 02:47:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.7010 [2026-10-20 02:47:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4289 [2026-10-20 02:47:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8179 [2026-10-20 02:47:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.5443 [2026-10-20 02:47:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8715 [2026-10-20 02:48:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1321 [2026-10-20 02:48:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.2212 [2026-10-20 02:48:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.9556 [2026-10-20 02:48:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.3647 [2026-10-20 02:48:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1593 [2026-10-20 02:48:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8590 [2026-10-20 02:48:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1941 [2026-10-20 02:48:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2001 [2026-10-20 02:48:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.9191 [2026-10-20 02:48:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.3233 [2026-10-20 02:48:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.6330 [2026-10-20 02:48:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0458 [2026-10-20 02:48:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3282 [2026-10-20 02:48:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8979 [2026-10-20 02:48:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8003 [2026-10-20 02:48:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8020 [2026-10-20 02:49:08][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,575 | MFU: 1.8% [2026-10-20 02:49:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3767 [2026-10-20 02:49:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4559 [2026-10-20 02:49:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.7722 [2026-10-20 02:49:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3550 [2026-10-20 02:49:12][INFO][training.pretrain_engine] [Ph2|Step 204] loss=7.2062 | ppl=1347.80 | aux=0.02516 | lr=1.49e-04 | gnorm=0.758 | tok/s=29,279 | MFU=7.0% | step=143251ms | tokens=0.221B | mem=4.2/32.7GB | cost=$107.12/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 23:01 (140.2h) | Phase3: SKIPPED | SFT: Oct 26 03:46 (145.0h) | DPO (Final): Oct 26 04:20 (145.5h) [2026-10-20 02:49:12][INFO][training.pretrain_engine] Starting global step 205 (accumulating 32 micro-batches)... [2026-10-20 02:49:12][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 02:49:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.4721 [2026-10-20 02:49:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2656 [2026-10-20 02:49:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1406 [2026-10-20 02:49:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0801 [2026-10-20 02:49:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1363 [2026-10-20 02:49:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1931 [2026-10-20 02:49:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2227 [2026-10-20 02:49:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9545 [2026-10-20 02:50:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1309 [2026-10-20 02:50:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.6282 [2026-10-20 02:50:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1950 [2026-10-20 02:50:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4297 [2026-10-20 02:50:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0349 [2026-10-20 02:50:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0823 [2026-10-20 02:50:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9657 [2026-10-20 02:50:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0404 [2026-10-20 02:50:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0879 [2026-10-20 02:50:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.4570 [2026-10-20 02:50:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.4953 [2026-10-20 02:50:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.9222 [2026-10-20 02:50:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.3787 [2026-10-20 02:50:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.4346 [2026-10-20 02:50:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9278 [2026-10-20 02:50:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7592 [2026-10-20 02:51:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3680 [2026-10-20 02:51:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2722 [2026-10-20 02:51:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.4366 [2026-10-20 02:51:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3614 [2026-10-20 02:51:30][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,601 | MFU: 1.8% [2026-10-20 02:51:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3394 [2026-10-20 02:51:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.5476 [2026-10-20 02:51:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.1724 [2026-10-20 02:51:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.9441 [2026-10-20 02:51:35][INFO][training.pretrain_engine] [Ph2|Step 205] loss=7.2317 | ppl=1382.57 | aux=0.02515 | lr=1.50e-04 | gnorm=1.180 | tok/s=29,281 | MFU=7.0% | step=143245ms | tokens=0.222B | mem=4.2/32.7GB | cost=$107.64/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 23:03 (140.2h) | Phase3: SKIPPED | SFT: Oct 26 03:47 (144.9h) | DPO (Final): Oct 26 04:21 (145.5h) [2026-10-20 02:51:35][INFO][training.pretrain_engine] Starting global step 206 (accumulating 32 micro-batches)... [2026-10-20 02:51:35][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 02:51:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.2419 [2026-10-20 02:51:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.0093 [2026-10-20 02:51:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.0427 [2026-10-20 02:51:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.0477 [2026-10-20 02:52:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2317 [2026-10-20 02:52:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2721 [2026-10-20 02:52:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9818 [2026-10-20 02:52:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4133 [2026-10-20 02:52:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9585 [2026-10-20 02:52:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8582 [2026-10-20 02:52:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8365 [2026-10-20 02:52:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.2851 [2026-10-20 02:52:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4325 [2026-10-20 02:52:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1690 [2026-10-20 02:52:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.5281 [2026-10-20 02:52:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1464 [2026-10-20 02:53:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7647 [2026-10-20 02:53:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.6131 [2026-10-20 02:53:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3603 [2026-10-20 02:53:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.4771 [2026-10-20 02:53:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.3432 [2026-10-20 02:53:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.4800 [2026-10-20 02:53:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.3674 [2026-10-20 02:53:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.3459 [2026-10-20 02:53:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.5866 [2026-10-20 02:53:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.9022 [2026-10-20 02:53:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.6999 [2026-10-20 02:53:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.6854 [2026-10-20 02:53:53][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,580 | MFU: 1.8% [2026-10-20 02:53:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.6209 [2026-10-20 02:53:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3870 [2026-10-20 02:53:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2985 [2026-10-20 02:53:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.1436 [2026-10-20 02:53:58][INFO][training.pretrain_engine] [Ph2|Step 206] loss=7.3771 | ppl=1598.91 | aux=0.02516 | lr=1.51e-04 | gnorm=0.930 | tok/s=29,281 | MFU=7.0% | step=143241ms | tokens=0.223B | mem=4.2/32.7GB | cost=$108.16/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 23:04 (140.2h) | Phase3: SKIPPED | SFT: Oct 26 03:49 (144.9h) | DPO (Final): Oct 26 04:23 (145.5h) [2026-10-20 02:53:58][INFO][training.pretrain_engine] Starting global step 207 (accumulating 32 micro-batches)... [2026-10-20 02:53:58][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 02:54:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0719 [2026-10-20 02:54:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1570 [2026-10-20 02:54:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8320 [2026-10-20 02:54:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9578 [2026-10-20 02:54:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.5133 [2026-10-20 02:54:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.6823 [2026-10-20 02:54:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.8144 [2026-10-20 02:54:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.6886 [2026-10-20 02:54:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5348 [2026-10-20 02:54:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4058 [2026-10-20 02:54:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.2697 [2026-10-20 02:54:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3405 [2026-10-20 02:55:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.7313 [2026-10-20 02:55:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.8287 [2026-10-20 02:55:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.0679 [2026-10-20 02:55:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.6795 [2026-10-20 02:55:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0156 [2026-10-20 02:55:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.9857 [2026-10-20 02:55:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.4154 [2026-10-20 02:55:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2317 [2026-10-20 02:55:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9514 [2026-10-20 02:55:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1747 [2026-10-20 02:55:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1577 [2026-10-20 02:55:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1265 [2026-10-20 02:56:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3778 [2026-10-20 02:56:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.4293 [2026-10-20 02:56:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3471 [2026-10-20 02:56:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2958 [2026-10-20 02:56:17][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,565 | MFU: 1.8% [2026-10-20 02:56:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2355 [2026-10-20 02:56:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8917 [2026-10-20 02:56:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.9664 [2026-10-20 02:56:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0246 [2026-10-20 02:56:21][INFO][training.pretrain_engine] [Ph2|Step 207] loss=7.3771 | ppl=1599.00 | aux=0.02519 | lr=1.51e-04 | gnorm=0.773 | tok/s=29,282 | MFU=7.0% | step=143238ms | tokens=0.224B | mem=4.2/32.7GB | cost=$108.69/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 23:06 (140.2h) | Phase3: SKIPPED | SFT: Oct 26 03:50 (144.9h) | DPO (Final): Oct 26 04:25 (145.5h) [2026-10-20 02:56:25][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step207 | Tokens: 0.224B | Spent: $108.70 | Remaining: $291.30 [2026-10-20 02:56:25][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step207 [2026-10-20 02:56:25][INFO][training.pretrain_engine] Starting global step 208 (accumulating 32 micro-batches)... [2026-10-20 02:56:25][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 02:56:25][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 02:56:25][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 02:56:25][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 02:56:25][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step207 to Phase-Technologies/forge-500m-pretrain (path: phase2_step207)... [2026-10-20 02:56:25][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 02:56:34][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 02:56:34][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step207 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 02:56:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.3557 [2026-10-20 02:56:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2222 [2026-10-20 02:56:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.6764 [2026-10-20 02:56:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2721 [2026-10-20 02:56:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.3171 [2026-10-20 02:56:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1007 [2026-10-20 02:56:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4759 [2026-10-20 02:57:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1797 [2026-10-20 02:57:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3747 [2026-10-20 02:57:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.8745 [2026-10-20 02:57:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.6089 [2026-10-20 02:57:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.2708 [2026-10-20 02:57:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2311 [2026-10-20 02:57:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1994 [2026-10-20 02:57:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2533 [2026-10-20 02:57:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.7501 [2026-10-20 02:57:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2512 [2026-10-20 02:57:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.4506 [2026-10-20 02:57:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1662 [2026-10-20 02:57:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0328 [2026-10-20 02:58:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.4858 [2026-10-20 02:58:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1598 [2026-10-20 02:58:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0509 [2026-10-20 02:58:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0027 [2026-10-20 02:58:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2267 [2026-10-20 02:58:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9511 [2026-10-20 02:58:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.8721 [2026-10-20 02:58:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.7522 [2026-10-20 02:58:43][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,558 | MFU: 1.8% [2026-10-20 02:58:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.0653 [2026-10-20 02:58:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.0317 [2026-10-20 02:58:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.9595 [2026-10-20 02:58:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.0453 [2026-10-20 02:58:48][INFO][training.pretrain_engine] [Ph2|Step 208] loss=7.3597 | ppl=1571.33 | aux=0.02518 | lr=1.52e-04 | gnorm=0.855 | tok/s=29,281 | MFU=7.0% | step=143241ms | tokens=0.225B | mem=4.2/32.7GB | cost=$109.22/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 23:08 (140.2h) | Phase3: SKIPPED | SFT: Oct 26 03:52 (144.9h) | DPO (Final): Oct 26 04:27 (145.5h) [2026-10-20 02:58:48][INFO][training.pretrain_engine] Starting global step 209 (accumulating 32 micro-batches)... [2026-10-20 02:58:48][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 02:59:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.6782 [2026-10-20 02:59:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.6808 [2026-10-20 02:59:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.8533 [2026-10-20 02:59:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.8717 [2026-10-20 02:59:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.6083 [2026-10-20 02:59:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4819 [2026-10-20 02:59:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.5851 [2026-10-20 02:59:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.3949 [2026-10-20 02:59:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0375 [2026-10-20 02:59:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8772 [2026-10-20 02:59:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9136 [2026-10-20 02:59:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0638 [2026-10-20 02:59:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.3441 [2026-10-20 02:59:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.7190 [2026-10-20 02:59:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.0536 [2026-10-20 02:59:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.6127 [2026-10-20 03:00:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.6384 [2026-10-20 03:00:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.9586 [2026-10-20 03:00:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.7468 [2026-10-20 03:00:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.7735 [2026-10-20 03:00:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0058 [2026-10-20 03:00:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2566 [2026-10-20 03:00:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2608 [2026-10-20 03:00:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7364 [2026-10-20 03:00:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2456 [2026-10-20 03:00:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.8282 [2026-10-20 03:00:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.2423 [2026-10-20 03:00:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1569 [2026-10-20 03:01:06][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,600 | MFU: 1.8% [2026-10-20 03:01:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2444 [2026-10-20 03:01:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2117 [2026-10-20 03:01:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9362 [2026-10-20 03:01:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.1725 [2026-10-20 03:01:11][INFO][training.pretrain_engine] [Ph2|Step 209] loss=7.3509 | ppl=1557.59 | aux=0.02518 | lr=1.52e-04 | gnorm=1.102 | tok/s=29,283 | MFU=7.0% | step=143234ms | tokens=0.226B | mem=4.2/32.7GB | cost=$109.74/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 23:09 (140.1h) | Phase3: SKIPPED | SFT: Oct 26 03:54 (144.9h) | DPO (Final): Oct 26 04:28 (145.5h) [2026-10-20 03:01:11][INFO][training.pretrain_engine] Starting global step 210 (accumulating 32 micro-batches)... [2026-10-20 03:01:11][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 03:01:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2723 [2026-10-20 03:01:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1497 [2026-10-20 03:01:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1141 [2026-10-20 03:01:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1452 [2026-10-20 03:01:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1272 [2026-10-20 03:01:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.3622 [2026-10-20 03:01:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2176 [2026-10-20 03:01:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.6080 [2026-10-20 03:02:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1627 [2026-10-20 03:02:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0102 [2026-10-20 03:02:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3858 [2026-10-20 03:02:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1160 [2026-10-20 03:02:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0891 [2026-10-20 03:02:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.7577 [2026-10-20 03:02:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.5230 [2026-10-20 03:02:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8250 [2026-10-20 03:02:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1059 [2026-10-20 03:02:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.6253 [2026-10-20 03:02:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.7625 [2026-10-20 03:02:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3981 [2026-10-20 03:02:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.5515 [2026-10-20 03:02:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2701 [2026-10-20 03:02:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2483 [2026-10-20 03:02:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1072 [2026-10-20 03:03:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0858 [2026-10-20 03:03:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2266 [2026-10-20 03:03:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1802 [2026-10-20 03:03:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2587 [2026-10-20 03:03:29][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,605 | MFU: 1.8% [2026-10-20 03:03:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.1283 [2026-10-20 03:03:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3254 [2026-10-20 03:03:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6718 [2026-10-20 03:03:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0679 [2026-10-20 03:03:34][INFO][training.pretrain_engine] [Ph2|Step 210] loss=7.3219 | ppl=1513.11 | aux=0.02511 | lr=1.53e-04 | gnorm=0.855 | tok/s=29,284 | MFU=7.0% | step=143231ms | tokens=0.228B | mem=4.2/32.7GB | cost=$110.27/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 23:11 (140.1h) | Phase3: SKIPPED | SFT: Oct 26 03:55 (144.9h) | DPO (Final): Oct 26 04:30 (145.4h) [2026-10-20 03:03:34][INFO][training.pretrain_engine] Starting global step 211 (accumulating 32 micro-batches)... [2026-10-20 03:03:34][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 03:03:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0141 [2026-10-20 03:03:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1343 [2026-10-20 03:03:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.3026 [2026-10-20 03:03:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.3592 [2026-10-20 03:04:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4404 [2026-10-20 03:04:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2527 [2026-10-20 03:04:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.3534 [2026-10-20 03:04:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1790 [2026-10-20 03:04:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4447 [2026-10-20 03:04:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8823 [2026-10-20 03:04:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.5407 [2026-10-20 03:04:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.2399 [2026-10-20 03:04:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.3364 [2026-10-20 03:04:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4465 [2026-10-20 03:04:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9408 [2026-10-20 03:04:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2175 [2026-10-20 03:05:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.6003 [2026-10-20 03:05:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2146 [2026-10-20 03:05:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1192 [2026-10-20 03:05:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3828 [2026-10-20 03:05:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9735 [2026-10-20 03:05:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.3193 [2026-10-20 03:05:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2670 [2026-10-20 03:05:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.5801 [2026-10-20 03:05:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3779 [2026-10-20 03:05:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2547 [2026-10-20 03:05:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3523 [2026-10-20 03:05:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3681 [2026-10-20 03:05:51][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,621 | MFU: 1.8% [2026-10-20 03:05:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2713 [2026-10-20 03:05:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.5810 [2026-10-20 03:05:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4630 [2026-10-20 03:05:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4982 [2026-10-20 03:05:57][INFO][training.pretrain_engine] [Ph2|Step 211] loss=7.3019 | ppl=1483.06 | aux=0.02513 | lr=1.53e-04 | gnorm=0.844 | tok/s=29,284 | MFU=7.0% | step=143229ms | tokens=0.229B | mem=4.2/32.7GB | cost=$110.79/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 23:12 (140.1h) | Phase3: SKIPPED | SFT: Oct 26 03:57 (144.9h) | DPO (Final): Oct 26 04:31 (145.4h) [2026-10-20 03:05:57][INFO][training.pretrain_engine] Starting global step 212 (accumulating 32 micro-batches)... [2026-10-20 03:05:57][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 03:06:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.4345 [2026-10-20 03:06:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.6456 [2026-10-20 03:06:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9991 [2026-10-20 03:06:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1073 [2026-10-20 03:06:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0568 [2026-10-20 03:06:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4541 [2026-10-20 03:06:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2111 [2026-10-20 03:06:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.3090 [2026-10-20 03:06:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1836 [2026-10-20 03:06:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.2275 [2026-10-20 03:06:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1330 [2026-10-20 03:06:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0457 [2026-10-20 03:07:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.5381 [2026-10-20 03:07:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9750 [2026-10-20 03:07:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4593 [2026-10-20 03:07:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9603 [2026-10-20 03:07:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.7706 [2026-10-20 03:07:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.0398 [2026-10-20 03:07:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.1134 [2026-10-20 03:07:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.4954 [2026-10-20 03:07:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0549 [2026-10-20 03:07:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.6257 [2026-10-20 03:07:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0353 [2026-10-20 03:07:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0748 [2026-10-20 03:08:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9623 [2026-10-20 03:08:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.8736 [2026-10-20 03:08:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.7730 [2026-10-20 03:08:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1767 [2026-10-20 03:08:14][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,623 | MFU: 1.8% [2026-10-20 03:08:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4133 [2026-10-20 03:08:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.9373 [2026-10-20 03:08:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2979 [2026-10-20 03:08:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.8036 [2026-10-20 03:08:20][INFO][training.pretrain_engine] [Ph2|Step 212] loss=7.3856 | ppl=1612.65 | aux=0.02517 | lr=1.54e-04 | gnorm=1.016 | tok/s=29,285 | MFU=7.0% | step=143226ms | tokens=0.230B | mem=4.2/32.7GB | cost=$111.31/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 23:14 (140.1h) | Phase3: SKIPPED | SFT: Oct 26 03:59 (144.8h) | DPO (Final): Oct 26 04:33 (145.4h) [2026-10-20 03:08:20][INFO][training.pretrain_engine] Starting global step 213 (accumulating 32 micro-batches)... [2026-10-20 03:08:20][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 03:08:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9526 [2026-10-20 03:08:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8401 [2026-10-20 03:08:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.5697 [2026-10-20 03:08:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0481 [2026-10-20 03:08:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0473 [2026-10-20 03:08:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0081 [2026-10-20 03:08:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8317 [2026-10-20 03:08:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9961 [2026-10-20 03:09:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4860 [2026-10-20 03:09:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.6608 [2026-10-20 03:09:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.7568 [2026-10-20 03:09:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.0945 [2026-10-20 03:09:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1130 [2026-10-20 03:09:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.5162 [2026-10-20 03:09:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0060 [2026-10-20 03:09:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.3693 [2026-10-20 03:09:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.5981 [2026-10-20 03:09:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2552 [2026-10-20 03:09:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9475 [2026-10-20 03:09:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.6768 [2026-10-20 03:10:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0132 [2026-10-20 03:10:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.5169 [2026-10-20 03:10:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0648 [2026-10-20 03:10:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7564 [2026-10-20 03:10:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3372 [2026-10-20 03:10:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0774 [2026-10-20 03:10:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0821 [2026-10-20 03:10:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9663 [2026-10-20 03:10:38][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,591 | MFU: 1.8% [2026-10-20 03:10:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.1656 [2026-10-20 03:10:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.6951 [2026-10-20 03:10:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4623 [2026-10-20 03:10:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3479 [2026-10-20 03:10:43][INFO][training.pretrain_engine] [Ph2|Step 213] loss=7.2862 | ppl=1460.06 | aux=0.02512 | lr=1.55e-04 | gnorm=1.203 | tok/s=29,285 | MFU=7.0% | step=143224ms | tokens=0.231B | mem=4.2/32.7GB | cost=$111.84/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 23:16 (140.1h) | Phase3: SKIPPED | SFT: Oct 26 04:00 (144.8h) | DPO (Final): Oct 26 04:34 (145.4h) [2026-10-20 03:10:43][INFO][training.pretrain_engine] Starting global step 214 (accumulating 32 micro-batches)... [2026-10-20 03:10:43][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 03:11:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2842 [2026-10-20 03:11:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0686 [2026-10-20 03:11:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.3702 [2026-10-20 03:11:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.4166 [2026-10-20 03:11:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1939 [2026-10-20 03:11:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4508 [2026-10-20 03:11:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.6874 [2026-10-20 03:11:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.3864 [2026-10-20 03:11:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1343 [2026-10-20 03:11:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1201 [2026-10-20 03:11:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4404 [2026-10-20 03:11:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1101 [2026-10-20 03:11:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4419 [2026-10-20 03:11:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2463 [2026-10-20 03:11:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8589 [2026-10-20 03:11:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9956 [2026-10-20 03:12:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1796 [2026-10-20 03:12:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.6075 [2026-10-20 03:12:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0661 [2026-10-20 03:12:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3421 [2026-10-20 03:12:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.5946 [2026-10-20 03:12:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.6232 [2026-10-20 03:12:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.9318 [2026-10-20 03:12:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.4792 [2026-10-20 03:12:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.4131 [2026-10-20 03:12:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1821 [2026-10-20 03:12:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3600 [2026-10-20 03:12:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3269 [2026-10-20 03:13:01][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,571 | MFU: 1.8% [2026-10-20 03:13:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4541 [2026-10-20 03:13:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4421 [2026-10-20 03:13:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.9735 [2026-10-20 03:13:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8979 [2026-10-20 03:13:06][INFO][training.pretrain_engine] [Ph2|Step 214] loss=7.1687 | ppl=1298.19 | aux=0.02517 | lr=1.55e-04 | gnorm=0.992 | tok/s=29,286 | MFU=7.0% | step=143220ms | tokens=0.232B | mem=4.2/32.7GB | cost=$112.36/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 23:17 (140.1h) | Phase3: SKIPPED | SFT: Oct 26 04:02 (144.8h) | DPO (Final): Oct 26 04:36 (145.4h) [2026-10-20 03:13:06][INFO][training.pretrain_engine] Starting global step 215 (accumulating 32 micro-batches)... [2026-10-20 03:13:06][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 03:13:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.3179 [2026-10-20 03:13:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0381 [2026-10-20 03:13:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0055 [2026-10-20 03:13:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0685 [2026-10-20 03:13:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2197 [2026-10-20 03:13:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1594 [2026-10-20 03:13:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6484 [2026-10-20 03:13:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8179 [2026-10-20 03:13:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.0183 [2026-10-20 03:13:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3022 [2026-10-20 03:13:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4970 [2026-10-20 03:14:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.1653 [2026-10-20 03:14:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8297 [2026-10-20 03:14:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4277 [2026-10-20 03:14:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7387 [2026-10-20 03:14:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2024 [2026-10-20 03:14:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.8057 [2026-10-20 03:14:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3425 [2026-10-20 03:14:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3173 [2026-10-20 03:14:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3699 [2026-10-20 03:14:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.5312 [2026-10-20 03:14:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1695 [2026-10-20 03:14:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.6491 [2026-10-20 03:14:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9372 [2026-10-20 03:15:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.6988 [2026-10-20 03:15:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.8679 [2026-10-20 03:15:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2485 [2026-10-20 03:15:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.5889 [2026-10-20 03:15:25][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,554 | MFU: 1.8% [2026-10-20 03:15:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0622 [2026-10-20 03:15:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3089 [2026-10-20 03:15:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3990 [2026-10-20 03:15:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3115 [2026-10-20 03:15:29][INFO][training.pretrain_engine] [Ph2|Step 215] loss=7.3127 | ppl=1499.27 | aux=0.02517 | lr=1.56e-04 | gnorm=1.125 | tok/s=29,285 | MFU=7.0% | step=143221ms | tokens=0.233B | mem=4.2/32.7GB | cost=$112.88/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 23:19 (140.1h) | Phase3: SKIPPED | SFT: Oct 26 04:04 (144.8h) | DPO (Final): Oct 26 04:38 (145.4h) [2026-10-20 03:15:29][INFO][training.pretrain_engine] Starting global step 216 (accumulating 32 micro-batches)... [2026-10-20 03:15:29][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 03:15:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2919 [2026-10-20 03:15:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9546 [2026-10-20 03:15:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0732 [2026-10-20 03:15:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.3110 [2026-10-20 03:16:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1247 [2026-10-20 03:16:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4504 [2026-10-20 03:16:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.3713 [2026-10-20 03:16:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2540 [2026-10-20 03:16:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.2459 [2026-10-20 03:16:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.2466 [2026-10-20 03:16:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.2384 [2026-10-20 03:16:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.5414 [2026-10-20 03:16:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4037 [2026-10-20 03:16:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2487 [2026-10-20 03:16:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2289 [2026-10-20 03:16:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.8046 [2026-10-20 03:16:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1296 [2026-10-20 03:16:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9373 [2026-10-20 03:16:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6877 [2026-10-20 03:16:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3092 [2026-10-20 03:17:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0743 [2026-10-20 03:17:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1133 [2026-10-20 03:17:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2207 [2026-10-20 03:17:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.4299 [2026-10-20 03:17:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.5842 [2026-10-20 03:17:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.6617 [2026-10-20 03:17:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2821 [2026-10-20 03:17:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2345 [2026-10-20 03:17:47][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,603 | MFU: 1.8% [2026-10-20 03:17:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.8738 [2026-10-20 03:17:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.7034 [2026-10-20 03:17:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.3834 [2026-10-20 03:17:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.0418 [2026-10-20 03:17:52][INFO][training.pretrain_engine] [Ph2|Step 216] loss=7.4589 | ppl=1735.30 | aux=0.02520 | lr=1.56e-04 | gnorm=0.988 | tok/s=29,285 | MFU=7.0% | step=143223ms | tokens=0.234B | mem=4.2/32.7GB | cost=$113.41/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 23:21 (140.1h) | Phase3: SKIPPED | SFT: Oct 26 04:06 (144.8h) | DPO (Final): Oct 26 04:40 (145.4h) [2026-10-20 03:17:56][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step216 | Tokens: 0.234B | Spent: $113.42 | Remaining: $286.58 [2026-10-20 03:17:56][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step216 [2026-10-20 03:17:56][INFO][training.pretrain_engine] Starting global step 217 (accumulating 32 micro-batches)... [2026-10-20 03:17:56][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 03:17:56][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 03:17:56][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 03:17:56][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 03:17:56][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step216 to Phase-Technologies/forge-500m-pretrain (path: phase2_step216)... [2026-10-20 03:17:56][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 03:18:06][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 03:18:06][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step216 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 03:18:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9763 [2026-10-20 03:18:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8458 [2026-10-20 03:18:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2131 [2026-10-20 03:18:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9272 [2026-10-20 03:18:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.5096 [2026-10-20 03:18:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6513 [2026-10-20 03:18:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.3789 [2026-10-20 03:18:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2478 [2026-10-20 03:18:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.5345 [2026-10-20 03:18:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.0372 [2026-10-20 03:18:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.6359 [2026-10-20 03:18:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4428 [2026-10-20 03:19:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8432 [2026-10-20 03:19:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7646 [2026-10-20 03:19:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7785 [2026-10-20 03:19:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1257 [2026-10-20 03:19:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.7651 [2026-10-20 03:19:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1509 [2026-10-20 03:19:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0906 [2026-10-20 03:19:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3927 [2026-10-20 03:19:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.8464 [2026-10-20 03:19:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2335 [2026-10-20 03:19:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9846 [2026-10-20 03:19:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.9799 [2026-10-20 03:19:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0651 [2026-10-20 03:19:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1358 [2026-10-20 03:19:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.5565 [2026-10-20 03:20:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1373 [2026-10-20 03:20:14][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,605 | MFU: 1.8% [2026-10-20 03:20:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0548 [2026-10-20 03:20:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.5475 [2026-10-20 03:20:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2613 [2026-10-20 03:20:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.1920 [2026-10-20 03:20:18][INFO][training.pretrain_engine] [Ph2|Step 217] loss=7.2699 | ppl=1436.36 | aux=0.02515 | lr=1.57e-04 | gnorm=0.691 | tok/s=29,287 | MFU=7.0% | step=143213ms | tokens=0.235B | mem=4.2/32.7GB | cost=$113.94/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 23:22 (140.0h) | Phase3: SKIPPED | SFT: Oct 26 04:07 (144.8h) | DPO (Final): Oct 26 04:41 (145.4h) [2026-10-20 03:20:18][INFO][training.pretrain_engine] Starting global step 218 (accumulating 32 micro-batches)... [2026-10-20 03:20:18][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 03:20:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.2535 [2026-10-20 03:20:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.3957 [2026-10-20 03:20:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.9370 [2026-10-20 03:20:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1153 [2026-10-20 03:20:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1090 [2026-10-20 03:20:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4962 [2026-10-20 03:20:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2229 [2026-10-20 03:20:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0120 [2026-10-20 03:21:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0122 [2026-10-20 03:21:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0045 [2026-10-20 03:21:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3689 [2026-10-20 03:21:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.2992 [2026-10-20 03:21:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1744 [2026-10-20 03:21:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9581 [2026-10-20 03:21:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9418 [2026-10-20 03:21:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1295 [2026-10-20 03:21:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9601 [2026-10-20 03:21:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3405 [2026-10-20 03:21:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0881 [2026-10-20 03:21:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1877 [2026-10-20 03:22:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0010 [2026-10-20 03:22:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8099 [2026-10-20 03:22:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8368 [2026-10-20 03:22:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1588 [2026-10-20 03:22:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3533 [2026-10-20 03:22:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8879 [2026-10-20 03:22:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9145 [2026-10-20 03:22:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7619 [2026-10-20 03:22:36][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,577 | MFU: 1.8% [2026-10-20 03:22:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9369 [2026-10-20 03:22:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.5807 [2026-10-20 03:22:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0049 [2026-10-20 03:22:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.8602 [2026-10-20 03:22:41][INFO][training.pretrain_engine] [Ph2|Step 218] loss=7.1196 | ppl=1235.94 | aux=0.02511 | lr=1.57e-04 | gnorm=0.891 | tok/s=29,288 | MFU=7.0% | step=143211ms | tokens=0.236B | mem=4.2/32.7GB | cost=$114.46/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 23:24 (140.0h) | Phase3: SKIPPED | SFT: Oct 26 04:09 (144.8h) | DPO (Final): Oct 26 04:43 (145.3h) [2026-10-20 03:22:41][INFO][training.pretrain_engine] Starting global step 219 (accumulating 32 micro-batches)... [2026-10-20 03:22:41][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 03:22:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7484 [2026-10-20 03:22:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8206 [2026-10-20 03:22:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0083 [2026-10-20 03:22:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8782 [2026-10-20 03:23:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.7103 [2026-10-20 03:23:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.6754 [2026-10-20 03:23:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.3501 [2026-10-20 03:23:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2891 [2026-10-20 03:23:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4774 [2026-10-20 03:23:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8460 [2026-10-20 03:23:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.2906 [2026-10-20 03:23:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1290 [2026-10-20 03:23:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1132 [2026-10-20 03:23:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9142 [2026-10-20 03:23:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8989 [2026-10-20 03:23:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9660 [2026-10-20 03:24:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0681 [2026-10-20 03:24:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0454 [2026-10-20 03:24:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1187 [2026-10-20 03:24:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2800 [2026-10-20 03:24:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1099 [2026-10-20 03:24:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9104 [2026-10-20 03:24:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9759 [2026-10-20 03:24:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8920 [2026-10-20 03:24:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1864 [2026-10-20 03:24:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0528 [2026-10-20 03:24:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1130 [2026-10-20 03:24:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3275 [2026-10-20 03:24:59][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,609 | MFU: 1.8% [2026-10-20 03:25:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.1570 [2026-10-20 03:25:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9006 [2026-10-20 03:25:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2010 [2026-10-20 03:25:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0794 [2026-10-20 03:25:04][INFO][training.pretrain_engine] [Ph2|Step 219] loss=7.2196 | ppl=1365.97 | aux=0.02516 | lr=1.58e-04 | gnorm=1.219 | tok/s=29,287 | MFU=7.0% | step=143214ms | tokens=0.237B | mem=4.2/32.7GB | cost=$114.98/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 23:26 (140.0h) | Phase3: SKIPPED | SFT: Oct 26 04:10 (144.8h) | DPO (Final): Oct 26 04:45 (145.3h) [2026-10-20 03:25:04][INFO][training.pretrain_engine] Starting global step 220 (accumulating 32 micro-batches)... [2026-10-20 03:25:04][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 03:25:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0294 [2026-10-20 03:25:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1325 [2026-10-20 03:25:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8789 [2026-10-20 03:25:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9209 [2026-10-20 03:25:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9557 [2026-10-20 03:25:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.3153 [2026-10-20 03:25:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.6614 [2026-10-20 03:25:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4662 [2026-10-20 03:25:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8530 [2026-10-20 03:25:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1791 [2026-10-20 03:25:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1904 [2026-10-20 03:25:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1306 [2026-10-20 03:26:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0956 [2026-10-20 03:26:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0819 [2026-10-20 03:26:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1083 [2026-10-20 03:26:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2186 [2026-10-20 03:26:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9480 [2026-10-20 03:26:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2597 [2026-10-20 03:26:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8412 [2026-10-20 03:26:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1848 [2026-10-20 03:26:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.7142 [2026-10-20 03:26:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.7850 [2026-10-20 03:26:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6438 [2026-10-20 03:26:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9055 [2026-10-20 03:27:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.5943 [2026-10-20 03:27:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2801 [2026-10-20 03:27:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.6904 [2026-10-20 03:27:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1666 [2026-10-20 03:27:22][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,585 | MFU: 1.8% [2026-10-20 03:27:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9706 [2026-10-20 03:27:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.1833 [2026-10-20 03:27:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0399 [2026-10-20 03:27:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8983 [2026-10-20 03:27:27][INFO][training.pretrain_engine] [Ph2|Step 220] loss=7.1551 | ppl=1280.61 | aux=0.02512 | lr=1.59e-04 | gnorm=1.008 | tok/s=29,287 | MFU=7.0% | step=143215ms | tokens=0.238B | mem=4.2/32.7GB | cost=$115.51/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 23:28 (140.0h) | Phase3: SKIPPED | SFT: Oct 26 04:12 (144.8h) | DPO (Final): Oct 26 04:46 (145.3h) [2026-10-20 03:27:27][INFO][training.pretrain_engine] Starting global step 221 (accumulating 32 micro-batches)... [2026-10-20 03:27:27][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 03:27:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2230 [2026-10-20 03:27:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.3414 [2026-10-20 03:27:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0357 [2026-10-20 03:27:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1255 [2026-10-20 03:28:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0434 [2026-10-20 03:28:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0644 [2026-10-20 03:28:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0144 [2026-10-20 03:28:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4226 [2026-10-20 03:28:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.2417 [2026-10-20 03:28:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9921 [2026-10-20 03:28:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1943 [2026-10-20 03:28:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8904 [2026-10-20 03:28:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2274 [2026-10-20 03:28:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0671 [2026-10-20 03:28:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2505 [2026-10-20 03:28:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0878 [2026-10-20 03:28:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7874 [2026-10-20 03:28:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0493 [2026-10-20 03:28:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8569 [2026-10-20 03:28:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8922 [2026-10-20 03:29:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0141 [2026-10-20 03:29:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9409 [2026-10-20 03:29:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.4335 [2026-10-20 03:29:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6755 [2026-10-20 03:29:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2712 [2026-10-20 03:29:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9690 [2026-10-20 03:29:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1587 [2026-10-20 03:29:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8816 [2026-10-20 03:29:45][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,608 | MFU: 1.8% [2026-10-20 03:29:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.5553 [2026-10-20 03:29:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9819 [2026-10-20 03:29:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0019 [2026-10-20 03:29:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9597 [2026-10-20 03:29:50][INFO][training.pretrain_engine] [Ph2|Step 221] loss=7.1216 | ppl=1238.45 | aux=0.02516 | lr=1.59e-04 | gnorm=1.008 | tok/s=29,287 | MFU=7.0% | step=143214ms | tokens=0.239B | mem=4.2/32.7GB | cost=$116.03/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 23:30 (140.0h) | Phase3: SKIPPED | SFT: Oct 26 04:14 (144.7h) | DPO (Final): Oct 26 04:48 (145.3h) [2026-10-20 03:29:50][INFO][training.pretrain_engine] Starting global step 222 (accumulating 32 micro-batches)... [2026-10-20 03:29:50][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 03:30:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1083 [2026-10-20 03:30:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2043 [2026-10-20 03:30:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.4461 [2026-10-20 03:30:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0399 [2026-10-20 03:30:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1684 [2026-10-20 03:30:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4140 [2026-10-20 03:30:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9281 [2026-10-20 03:30:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2944 [2026-10-20 03:30:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.9511 [2026-10-20 03:30:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.2455 [2026-10-20 03:30:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7575 [2026-10-20 03:30:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.6154 [2026-10-20 03:31:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4230 [2026-10-20 03:31:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.2499 [2026-10-20 03:31:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.1442 [2026-10-20 03:31:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.7531 [2026-10-20 03:31:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8988 [2026-10-20 03:31:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9633 [2026-10-20 03:31:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9671 [2026-10-20 03:31:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7388 [2026-10-20 03:31:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9207 [2026-10-20 03:31:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9541 [2026-10-20 03:31:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.4324 [2026-10-20 03:31:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8837 [2026-10-20 03:31:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9488 [2026-10-20 03:31:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0267 [2026-10-20 03:31:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.5581 [2026-10-20 03:31:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1090 [2026-10-20 03:32:07][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,622 | MFU: 1.8% [2026-10-20 03:32:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3137 [2026-10-20 03:32:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3887 [2026-10-20 03:32:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.5531 [2026-10-20 03:32:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4011 [2026-10-20 03:32:13][INFO][training.pretrain_engine] [Ph2|Step 222] loss=7.1635 | ppl=1291.42 | aux=0.02514 | lr=1.60e-04 | gnorm=1.234 | tok/s=29,288 | MFU=7.0% | step=143211ms | tokens=0.240B | mem=4.2/32.7GB | cost=$116.55/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 23:31 (140.0h) | Phase3: SKIPPED | SFT: Oct 26 04:16 (144.7h) | DPO (Final): Oct 26 04:50 (145.3h) [2026-10-20 03:32:13][INFO][training.pretrain_engine] Starting global step 223 (accumulating 32 micro-batches)... [2026-10-20 03:32:13][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 03:32:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7556 [2026-10-20 03:32:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0501 [2026-10-20 03:32:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0422 [2026-10-20 03:32:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1257 [2026-10-20 03:32:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1894 [2026-10-20 03:32:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2373 [2026-10-20 03:32:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1373 [2026-10-20 03:32:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7973 [2026-10-20 03:33:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8932 [2026-10-20 03:33:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3208 [2026-10-20 03:33:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.5052 [2026-10-20 03:33:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4686 [2026-10-20 03:33:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1738 [2026-10-20 03:33:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0309 [2026-10-20 03:33:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8436 [2026-10-20 03:33:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0034 [2026-10-20 03:33:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=8.1704 [2026-10-20 03:33:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.8383 [2026-10-20 03:33:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.8414 [2026-10-20 03:33:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.7759 [2026-10-20 03:33:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.3841 [2026-10-20 03:33:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2072 [2026-10-20 03:33:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1296 [2026-10-20 03:34:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.3178 [2026-10-20 03:34:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1146 [2026-10-20 03:34:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2827 [2026-10-20 03:34:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.4330 [2026-10-20 03:34:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0646 [2026-10-20 03:34:30][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,613 | MFU: 1.8% [2026-10-20 03:34:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8085 [2026-10-20 03:34:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8740 [2026-10-20 03:34:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0832 [2026-10-20 03:34:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.1779 [2026-10-20 03:34:35][INFO][training.pretrain_engine] [Ph2|Step 223] loss=7.1366 | ppl=1257.16 | aux=0.02514 | lr=1.60e-04 | gnorm=0.852 | tok/s=29,289 | MFU=7.0% | step=143203ms | tokens=0.241B | mem=4.2/32.7GB | cost=$117.07/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 23:32 (140.0h) | Phase3: SKIPPED | SFT: Oct 26 04:17 (144.7h) | DPO (Final): Oct 26 04:51 (145.3h) [2026-10-20 03:34:35][INFO][training.pretrain_engine] Starting global step 224 (accumulating 32 micro-batches)... [2026-10-20 03:34:35][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 03:34:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8967 [2026-10-20 03:34:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7336 [2026-10-20 03:34:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8287 [2026-10-20 03:34:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5341 [2026-10-20 03:35:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.3357 [2026-10-20 03:35:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0975 [2026-10-20 03:35:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0360 [2026-10-20 03:35:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9981 [2026-10-20 03:35:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6683 [2026-10-20 03:35:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9761 [2026-10-20 03:35:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9617 [2026-10-20 03:35:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0587 [2026-10-20 03:35:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1757 [2026-10-20 03:35:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.5736 [2026-10-20 03:35:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4752 [2026-10-20 03:35:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6750 [2026-10-20 03:36:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6082 [2026-10-20 03:36:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9781 [2026-10-20 03:36:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8827 [2026-10-20 03:36:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9595 [2026-10-20 03:36:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1386 [2026-10-20 03:36:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1934 [2026-10-20 03:36:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2761 [2026-10-20 03:36:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2379 [2026-10-20 03:36:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.6987 [2026-10-20 03:36:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.5513 [2026-10-20 03:36:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.7505 [2026-10-20 03:36:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9941 [2026-10-20 03:36:54][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,577 | MFU: 1.8% [2026-10-20 03:36:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3941 [2026-10-20 03:36:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2890 [2026-10-20 03:36:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0678 [2026-10-20 03:36:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8923 [2026-10-20 03:36:58][INFO][training.pretrain_engine] [Ph2|Step 224] loss=7.0839 | ppl=1192.57 | aux=0.02518 | lr=1.61e-04 | gnorm=1.477 | tok/s=29,290 | MFU=7.0% | step=143197ms | tokens=0.242B | mem=4.2/32.7GB | cost=$117.59/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 23:34 (140.0h) | Phase3: SKIPPED | SFT: Oct 26 04:18 (144.7h) | DPO (Final): Oct 26 04:53 (145.3h) [2026-10-20 03:36:58][INFO][training.pretrain_engine] Starting global step 225 (accumulating 32 micro-batches)... [2026-10-20 03:36:58][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 03:37:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9470 [2026-10-20 03:37:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0718 [2026-10-20 03:37:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8176 [2026-10-20 03:37:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0237 [2026-10-20 03:37:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1243 [2026-10-20 03:37:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9655 [2026-10-20 03:37:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0665 [2026-10-20 03:37:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6090 [2026-10-20 03:37:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7443 [2026-10-20 03:37:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4160 [2026-10-20 03:37:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0597 [2026-10-20 03:37:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9244 [2026-10-20 03:38:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4888 [2026-10-20 03:38:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.3626 [2026-10-20 03:38:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7192 [2026-10-20 03:38:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2433 [2026-10-20 03:38:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0977 [2026-10-20 03:38:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0912 [2026-10-20 03:38:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3179 [2026-10-20 03:38:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.4895 [2026-10-20 03:38:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1047 [2026-10-20 03:38:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1075 [2026-10-20 03:38:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0047 [2026-10-20 03:38:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9760 [2026-10-20 03:39:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.5262 [2026-10-20 03:39:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7317 [2026-10-20 03:39:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.7225 [2026-10-20 03:39:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.8750 [2026-10-20 03:39:16][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,591 | MFU: 1.8% [2026-10-20 03:39:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.1639 [2026-10-20 03:39:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8431 [2026-10-20 03:39:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0980 [2026-10-20 03:39:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3945 [2026-10-20 03:39:21][INFO][training.pretrain_engine] [Ph2|Step 225] loss=7.1377 | ppl=1258.59 | aux=0.02513 | lr=1.61e-04 | gnorm=0.949 | tok/s=29,292 | MFU=7.0% | step=143189ms | tokens=0.243B | mem=4.2/32.7GB | cost=$118.12/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 23:35 (139.9h) | Phase3: SKIPPED | SFT: Oct 26 04:20 (144.7h) | DPO (Final): Oct 26 04:54 (145.2h) [2026-10-20 03:39:24][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step225 | Tokens: 0.243B | Spent: $118.13 | Remaining: $281.87 [2026-10-20 03:39:24][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step225 [2026-10-20 03:39:24][INFO][training.pretrain_engine] Starting global step 226 (accumulating 32 micro-batches)... [2026-10-20 03:39:24][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 03:39:24][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 03:39:24][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 03:39:24][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 03:39:24][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step225 to Phase-Technologies/forge-500m-pretrain (path: phase2_step225)... [2026-10-20 03:39:24][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 03:39:34][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 03:39:34][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step225 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 03:39:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.8588 [2026-10-20 03:39:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.5455 [2026-10-20 03:39:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.3750 [2026-10-20 03:39:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2057 [2026-10-20 03:39:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.0384 [2026-10-20 03:39:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.6987 [2026-10-20 03:39:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.3789 [2026-10-20 03:40:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4656 [2026-10-20 03:40:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7296 [2026-10-20 03:40:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.7912 [2026-10-20 03:40:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0048 [2026-10-20 03:40:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.8877 [2026-10-20 03:40:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2370 [2026-10-20 03:40:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.3232 [2026-10-20 03:40:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1659 [2026-10-20 03:40:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.5396 [2026-10-20 03:40:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9796 [2026-10-20 03:40:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9057 [2026-10-20 03:40:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0628 [2026-10-20 03:40:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.4710 [2026-10-20 03:41:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2005 [2026-10-20 03:41:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8656 [2026-10-20 03:41:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2513 [2026-10-20 03:41:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2162 [2026-10-20 03:41:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.5669 [2026-10-20 03:41:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.4361 [2026-10-20 03:41:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0653 [2026-10-20 03:41:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1423 [2026-10-20 03:41:42][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,604 | MFU: 1.8% [2026-10-20 03:41:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9560 [2026-10-20 03:41:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3429 [2026-10-20 03:41:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8926 [2026-10-20 03:41:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2277 [2026-10-20 03:41:46][INFO][training.pretrain_engine] [Ph2|Step 226] loss=7.2034 | ppl=1343.93 | aux=0.02516 | lr=1.62e-04 | gnorm=1.156 | tok/s=29,294 | MFU=7.0% | step=143180ms | tokens=0.244B | mem=4.2/32.7GB | cost=$118.65/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 23:36 (139.9h) | Phase3: SKIPPED | SFT: Oct 26 04:21 (144.7h) | DPO (Final): Oct 26 04:55 (145.2h) [2026-10-20 03:41:46][INFO][training.pretrain_engine] Starting global step 227 (accumulating 32 micro-batches)... [2026-10-20 03:41:46][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 03:42:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.8588 [2026-10-20 03:42:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.6056 [2026-10-20 03:42:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.4311 [2026-10-20 03:42:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.9537 [2026-10-20 03:42:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9918 [2026-10-20 03:42:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.6350 [2026-10-20 03:42:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.5801 [2026-10-20 03:42:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.5168 [2026-10-20 03:42:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.2667 [2026-10-20 03:42:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.2959 [2026-10-20 03:42:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0340 [2026-10-20 03:42:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4331 [2026-10-20 03:42:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.7304 [2026-10-20 03:42:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.5169 [2026-10-20 03:42:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4771 [2026-10-20 03:42:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.3525 [2026-10-20 03:43:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.5800 [2026-10-20 03:43:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.5084 [2026-10-20 03:43:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.7946 [2026-10-20 03:43:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.5767 [2026-10-20 03:43:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4553 [2026-10-20 03:43:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1374 [2026-10-20 03:43:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9356 [2026-10-20 03:43:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6700 [2026-10-20 03:43:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.7051 [2026-10-20 03:43:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.0305 [2026-10-20 03:43:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6476 [2026-10-20 03:43:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.7405 [2026-10-20 03:44:05][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,543 | MFU: 1.8% [2026-10-20 03:44:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9624 [2026-10-20 03:44:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0587 [2026-10-20 03:44:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9003 [2026-10-20 03:44:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2892 [2026-10-20 03:44:10][INFO][training.pretrain_engine] [Ph2|Step 227] loss=7.2870 | ppl=1461.14 | aux=0.02518 | lr=1.62e-04 | gnorm=0.750 | tok/s=29,293 | MFU=7.0% | step=143186ms | tokens=0.245B | mem=4.2/32.7GB | cost=$119.17/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 23:39 (139.9h) | Phase3: SKIPPED | SFT: Oct 26 04:23 (144.7h) | DPO (Final): Oct 26 04:57 (145.2h) [2026-10-20 03:44:10][INFO][training.pretrain_engine] Starting global step 228 (accumulating 32 micro-batches)... [2026-10-20 03:44:10][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 03:44:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.8686 [2026-10-20 03:44:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.3382 [2026-10-20 03:44:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.8116 [2026-10-20 03:44:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=8.4824 [2026-10-20 03:44:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0853 [2026-10-20 03:44:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.7087 [2026-10-20 03:44:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1808 [2026-10-20 03:44:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.5248 [2026-10-20 03:45:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3415 [2026-10-20 03:45:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.6469 [2026-10-20 03:45:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4739 [2026-10-20 03:45:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3554 [2026-10-20 03:45:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1025 [2026-10-20 03:45:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.3177 [2026-10-20 03:45:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6332 [2026-10-20 03:45:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.9604 [2026-10-20 03:45:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9426 [2026-10-20 03:45:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7429 [2026-10-20 03:45:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8464 [2026-10-20 03:45:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0490 [2026-10-20 03:45:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6372 [2026-10-20 03:45:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9432 [2026-10-20 03:45:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8009 [2026-10-20 03:45:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8226 [2026-10-20 03:46:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9901 [2026-10-20 03:46:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1443 [2026-10-20 03:46:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6810 [2026-10-20 03:46:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9809 [2026-10-20 03:46:28][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,585 | MFU: 1.8% [2026-10-20 03:46:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5830 [2026-10-20 03:46:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8005 [2026-10-20 03:46:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8364 [2026-10-20 03:46:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5788 [2026-10-20 03:46:33][INFO][training.pretrain_engine] [Ph2|Step 228] loss=7.1792 | ppl=1311.86 | aux=0.02517 | lr=1.63e-04 | gnorm=0.688 | tok/s=29,293 | MFU=7.0% | step=143185ms | tokens=0.246B | mem=4.2/32.7GB | cost=$119.70/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 23:40 (139.9h) | Phase3: SKIPPED | SFT: Oct 26 04:25 (144.6h) | DPO (Final): Oct 26 04:59 (145.2h) [2026-10-20 03:46:33][INFO][training.pretrain_engine] Starting global step 229 (accumulating 32 micro-batches)... [2026-10-20 03:46:33][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 03:46:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7368 [2026-10-20 03:46:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.3121 [2026-10-20 03:46:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.4491 [2026-10-20 03:46:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1693 [2026-10-20 03:47:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0096 [2026-10-20 03:47:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9995 [2026-10-20 03:47:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0364 [2026-10-20 03:47:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.3226 [2026-10-20 03:47:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6463 [2026-10-20 03:47:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6406 [2026-10-20 03:47:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7588 [2026-10-20 03:47:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9180 [2026-10-20 03:47:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.9239 [2026-10-20 03:47:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1395 [2026-10-20 03:47:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2428 [2026-10-20 03:47:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.9427 [2026-10-20 03:48:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4844 [2026-10-20 03:48:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9305 [2026-10-20 03:48:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0993 [2026-10-20 03:48:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3838 [2026-10-20 03:48:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.6540 [2026-10-20 03:48:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0349 [2026-10-20 03:48:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6777 [2026-10-20 03:48:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8277 [2026-10-20 03:48:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.8209 [2026-10-20 03:48:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8877 [2026-10-20 03:48:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.5787 [2026-10-20 03:48:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.6979 [2026-10-20 03:48:52][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,566 | MFU: 1.8% [2026-10-20 03:48:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8392 [2026-10-20 03:48:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8331 [2026-10-20 03:48:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7777 [2026-10-20 03:48:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4287 [2026-10-20 03:48:56][INFO][training.pretrain_engine] [Ph2|Step 229] loss=7.1353 | ppl=1255.47 | aux=0.02515 | lr=1.64e-04 | gnorm=1.039 | tok/s=29,293 | MFU=7.0% | step=143182ms | tokens=0.247B | mem=4.2/32.7GB | cost=$120.22/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 23:42 (139.9h) | Phase3: SKIPPED | SFT: Oct 26 04:26 (144.6h) | DPO (Final): Oct 26 05:01 (145.2h) [2026-10-20 03:48:56][INFO][training.pretrain_engine] Starting global step 230 (accumulating 32 micro-batches)... [2026-10-20 03:48:56][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 03:49:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.4181 [2026-10-20 03:49:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1424 [2026-10-20 03:49:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0503 [2026-10-20 03:49:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.4079 [2026-10-20 03:49:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.9571 [2026-10-20 03:49:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0325 [2026-10-20 03:49:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0295 [2026-10-20 03:49:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2540 [2026-10-20 03:49:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8827 [2026-10-20 03:49:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0982 [2026-10-20 03:49:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6399 [2026-10-20 03:49:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3542 [2026-10-20 03:50:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1718 [2026-10-20 03:50:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2183 [2026-10-20 03:50:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2852 [2026-10-20 03:50:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8984 [2026-10-20 03:50:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2545 [2026-10-20 03:50:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.4821 [2026-10-20 03:50:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.7202 [2026-10-20 03:50:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.5943 [2026-10-20 03:50:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1615 [2026-10-20 03:50:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2129 [2026-10-20 03:50:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1418 [2026-10-20 03:50:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.5219 [2026-10-20 03:51:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9695 [2026-10-20 03:51:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1914 [2026-10-20 03:51:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9189 [2026-10-20 03:51:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6413 [2026-10-20 03:51:14][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,580 | MFU: 1.8% [2026-10-20 03:51:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2654 [2026-10-20 03:51:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3498 [2026-10-20 03:51:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3605 [2026-10-20 03:51:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9259 [2026-10-20 03:51:19][INFO][training.pretrain_engine] [Ph2|Step 230] loss=7.1771 | ppl=1309.08 | aux=0.02518 | lr=1.64e-04 | gnorm=1.297 | tok/s=29,294 | MFU=7.0% | step=143182ms | tokens=0.249B | mem=4.2/32.7GB | cost=$120.74/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 23:44 (139.9h) | Phase3: SKIPPED | SFT: Oct 26 04:28 (144.6h) | DPO (Final): Oct 26 05:02 (145.2h) [2026-10-20 03:51:19][INFO][training.pretrain_engine] Starting global step 231 (accumulating 32 micro-batches)... [2026-10-20 03:51:19][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 03:51:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8694 [2026-10-20 03:51:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0128 [2026-10-20 03:51:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6204 [2026-10-20 03:51:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2047 [2026-10-20 03:51:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.5790 [2026-10-20 03:51:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6190 [2026-10-20 03:51:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5385 [2026-10-20 03:51:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6045 [2026-10-20 03:52:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1240 [2026-10-20 03:52:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0837 [2026-10-20 03:52:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1270 [2026-10-20 03:52:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9362 [2026-10-20 03:52:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0289 [2026-10-20 03:52:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.9291 [2026-10-20 03:52:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.7104 [2026-10-20 03:52:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1830 [2026-10-20 03:52:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2830 [2026-10-20 03:52:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3775 [2026-10-20 03:52:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9793 [2026-10-20 03:52:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.5811 [2026-10-20 03:53:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6978 [2026-10-20 03:53:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0363 [2026-10-20 03:53:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0952 [2026-10-20 03:53:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5119 [2026-10-20 03:53:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0136 [2026-10-20 03:53:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6540 [2026-10-20 03:53:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2057 [2026-10-20 03:53:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9224 [2026-10-20 03:53:37][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,595 | MFU: 1.8% [2026-10-20 03:53:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3169 [2026-10-20 03:53:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3066 [2026-10-20 03:53:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.1677 [2026-10-20 03:53:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.1732 [2026-10-20 03:53:42][INFO][training.pretrain_engine] [Ph2|Step 231] loss=7.0951 | ppl=1206.09 | aux=0.02513 | lr=1.65e-04 | gnorm=1.188 | tok/s=29,294 | MFU=7.0% | step=143180ms | tokens=0.250B | mem=4.2/32.7GB | cost=$121.26/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 23:45 (139.9h) | Phase3: SKIPPED | SFT: Oct 26 04:30 (144.6h) | DPO (Final): Oct 26 05:04 (145.2h) [2026-10-20 03:53:42][INFO][training.pretrain_engine] Starting global step 232 (accumulating 32 micro-batches)... [2026-10-20 03:53:42][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 03:53:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2603 [2026-10-20 03:54:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4547 [2026-10-20 03:54:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3227 [2026-10-20 03:54:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7682 [2026-10-20 03:54:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.5177 [2026-10-20 03:54:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0566 [2026-10-20 03:54:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4016 [2026-10-20 03:54:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9109 [2026-10-20 03:54:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0142 [2026-10-20 03:54:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0435 [2026-10-20 03:54:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0737 [2026-10-20 03:54:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.2064 [2026-10-20 03:54:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8013 [2026-10-20 03:54:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8474 [2026-10-20 03:54:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3750 [2026-10-20 03:54:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3740 [2026-10-20 03:55:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.4688 [2026-10-20 03:55:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.6306 [2026-10-20 03:55:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.7871 [2026-10-20 03:55:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0464 [2026-10-20 03:55:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2909 [2026-10-20 03:55:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7633 [2026-10-20 03:55:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6764 [2026-10-20 03:55:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9244 [2026-10-20 03:55:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1038 [2026-10-20 03:55:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7370 [2026-10-20 03:55:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2433 [2026-10-20 03:55:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8069 [2026-10-20 03:56:00][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,576 | MFU: 1.8% [2026-10-20 03:56:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3492 [2026-10-20 03:56:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9500 [2026-10-20 03:56:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0917 [2026-10-20 03:56:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2880 [2026-10-20 03:56:05][INFO][training.pretrain_engine] [Ph2|Step 232] loss=7.1179 | ppl=1233.82 | aux=0.02515 | lr=1.65e-04 | gnorm=0.730 | tok/s=29,295 | MFU=7.0% | step=143177ms | tokens=0.251B | mem=4.2/32.7GB | cost=$121.79/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 23:47 (139.9h) | Phase3: SKIPPED | SFT: Oct 26 04:31 (144.6h) | DPO (Final): Oct 26 05:06 (145.2h) [2026-10-20 03:56:05][INFO][training.pretrain_engine] Starting global step 233 (accumulating 32 micro-batches)... [2026-10-20 03:56:05][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 03:56:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1493 [2026-10-20 03:56:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9809 [2026-10-20 03:56:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0183 [2026-10-20 03:56:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8225 [2026-10-20 03:56:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9960 [2026-10-20 03:56:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0963 [2026-10-20 03:56:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9953 [2026-10-20 03:56:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6911 [2026-10-20 03:56:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3389 [2026-10-20 03:56:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9853 [2026-10-20 03:56:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5619 [2026-10-20 03:56:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0425 [2026-10-20 03:57:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4054 [2026-10-20 03:57:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.3839 [2026-10-20 03:57:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9726 [2026-10-20 03:57:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9754 [2026-10-20 03:57:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2636 [2026-10-20 03:57:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1408 [2026-10-20 03:57:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1199 [2026-10-20 03:57:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0072 [2026-10-20 03:57:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8864 [2026-10-20 03:57:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5134 [2026-10-20 03:57:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8406 [2026-10-20 03:57:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9530 [2026-10-20 03:58:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9073 [2026-10-20 03:58:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9481 [2026-10-20 03:58:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8274 [2026-10-20 03:58:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9712 [2026-10-20 03:58:23][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,606 | MFU: 1.8% [2026-10-20 03:58:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7981 [2026-10-20 03:58:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9435 [2026-10-20 03:58:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7378 [2026-10-20 03:58:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8327 [2026-10-20 03:58:28][INFO][training.pretrain_engine] [Ph2|Step 233] loss=7.1010 | ppl=1213.18 | aux=0.02514 | lr=1.66e-04 | gnorm=0.871 | tok/s=29,296 | MFU=7.0% | step=143170ms | tokens=0.252B | mem=4.2/32.7GB | cost=$122.31/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 23:48 (139.8h) | Phase3: SKIPPED | SFT: Oct 26 04:33 (144.6h) | DPO (Final): Oct 26 05:07 (145.2h) [2026-10-20 03:58:28][INFO][training.pretrain_engine] Starting global step 234 (accumulating 32 micro-batches)... [2026-10-20 03:58:28][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 03:58:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4173 [2026-10-20 03:58:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5404 [2026-10-20 03:58:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3162 [2026-10-20 03:58:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5844 [2026-10-20 03:59:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2016 [2026-10-20 03:59:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7622 [2026-10-20 03:59:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2449 [2026-10-20 03:59:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8740 [2026-10-20 03:59:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9183 [2026-10-20 03:59:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6630 [2026-10-20 03:59:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0806 [2026-10-20 03:59:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8068 [2026-10-20 03:59:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1991 [2026-10-20 03:59:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.6470 [2026-10-20 03:59:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=8.0186 [2026-10-20 03:59:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.3145 [2026-10-20 03:59:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5088 [2026-10-20 03:59:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3222 [2026-10-20 03:59:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5749 [2026-10-20 03:59:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6856 [2026-10-20 04:00:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.4770 [2026-10-20 04:00:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2158 [2026-10-20 04:00:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0435 [2026-10-20 04:00:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.8666 [2026-10-20 04:00:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9488 [2026-10-20 04:00:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9432 [2026-10-20 04:00:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0910 [2026-10-20 04:00:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8493 [2026-10-20 04:00:46][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,566 | MFU: 1.8% [2026-10-20 04:00:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.5216 [2026-10-20 04:00:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9806 [2026-10-20 04:00:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.1835 [2026-10-20 04:00:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2580 [2026-10-20 04:00:51][INFO][training.pretrain_engine] [Ph2|Step 234] loss=7.0027 | ppl=1099.57 | aux=0.02516 | lr=1.66e-04 | gnorm=1.180 | tok/s=29,296 | MFU=7.0% | step=143170ms | tokens=0.253B | mem=4.2/32.7GB | cost=$122.83/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 23:50 (139.8h) | Phase3: SKIPPED | SFT: Oct 26 04:35 (144.6h) | DPO (Final): Oct 26 05:09 (145.1h) [2026-10-20 04:00:54][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step234 | Tokens: 0.253B | Spent: $122.84 | Remaining: $277.16 [2026-10-20 04:00:54][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step234 [2026-10-20 04:00:54][INFO][training.pretrain_engine] Starting global step 235 (accumulating 32 micro-batches)... [2026-10-20 04:00:54][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 04:00:54][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 04:00:54][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 04:00:54][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 04:00:54][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step234 to Phase-Technologies/forge-500m-pretrain (path: phase2_step234)... [2026-10-20 04:00:54][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 04:01:03][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 04:01:03][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step234 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 04:01:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.4534 [2026-10-20 04:01:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8302 [2026-10-20 04:01:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0568 [2026-10-20 04:01:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1677 [2026-10-20 04:01:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8980 [2026-10-20 04:01:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4764 [2026-10-20 04:01:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.7172 [2026-10-20 04:01:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0457 [2026-10-20 04:01:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1138 [2026-10-20 04:01:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4535 [2026-10-20 04:01:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0524 [2026-10-20 04:01:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0495 [2026-10-20 04:02:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2037 [2026-10-20 04:02:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8708 [2026-10-20 04:02:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0422 [2026-10-20 04:02:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0942 [2026-10-20 04:02:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0087 [2026-10-20 04:02:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2465 [2026-10-20 04:02:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8892 [2026-10-20 04:02:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2833 [2026-10-20 04:02:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9303 [2026-10-20 04:02:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.7063 [2026-10-20 04:02:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0803 [2026-10-20 04:02:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.8228 [2026-10-20 04:02:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9326 [2026-10-20 04:02:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1305 [2026-10-20 04:02:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1324 [2026-10-20 04:02:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8719 [2026-10-20 04:03:13][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,574 | MFU: 1.8% [2026-10-20 04:03:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4914 [2026-10-20 04:03:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4016 [2026-10-20 04:03:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3869 [2026-10-20 04:03:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4821 [2026-10-20 04:03:17][INFO][training.pretrain_engine] [Ph2|Step 235] loss=7.1712 | ppl=1301.42 | aux=0.02515 | lr=1.67e-04 | gnorm=1.156 | tok/s=29,296 | MFU=7.0% | step=143169ms | tokens=0.254B | mem=4.2/32.7GB | cost=$123.37/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 23:52 (139.8h) | Phase3: SKIPPED | SFT: Oct 26 04:36 (144.6h) | DPO (Final): Oct 26 05:11 (145.1h) [2026-10-20 04:03:17][INFO][training.pretrain_engine] Starting global step 236 (accumulating 32 micro-batches)... [2026-10-20 04:03:17][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 04:03:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0710 [2026-10-20 04:03:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9717 [2026-10-20 04:03:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7593 [2026-10-20 04:03:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0163 [2026-10-20 04:03:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5393 [2026-10-20 04:03:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8590 [2026-10-20 04:03:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7136 [2026-10-20 04:03:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8088 [2026-10-20 04:04:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3752 [2026-10-20 04:04:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4088 [2026-10-20 04:04:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.6027 [2026-10-20 04:04:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.1639 [2026-10-20 04:04:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5260 [2026-10-20 04:04:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4805 [2026-10-20 04:04:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3613 [2026-10-20 04:04:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1446 [2026-10-20 04:04:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0576 [2026-10-20 04:04:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1290 [2026-10-20 04:04:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1758 [2026-10-20 04:04:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0118 [2026-10-20 04:05:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.5046 [2026-10-20 04:05:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9340 [2026-10-20 04:05:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8594 [2026-10-20 04:05:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9199 [2026-10-20 04:05:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1519 [2026-10-20 04:05:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.4973 [2026-10-20 04:05:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2524 [2026-10-20 04:05:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.5645 [2026-10-20 04:05:36][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,558 | MFU: 1.8% [2026-10-20 04:05:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2015 [2026-10-20 04:05:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2888 [2026-10-20 04:05:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9482 [2026-10-20 04:05:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7779 [2026-10-20 04:05:40][INFO][training.pretrain_engine] [Ph2|Step 236] loss=7.1411 | ppl=1262.86 | aux=0.02515 | lr=1.68e-04 | gnorm=0.934 | tok/s=29,296 | MFU=7.0% | step=143172ms | tokens=0.255B | mem=4.2/32.7GB | cost=$123.89/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 23:54 (139.8h) | Phase3: SKIPPED | SFT: Oct 26 04:38 (144.6h) | DPO (Final): Oct 26 05:13 (145.1h) [2026-10-20 04:05:40][INFO][training.pretrain_engine] Starting global step 237 (accumulating 32 micro-batches)... [2026-10-20 04:05:40][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 04:05:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9443 [2026-10-20 04:05:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8975 [2026-10-20 04:05:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8670 [2026-10-20 04:05:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7969 [2026-10-20 04:06:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9182 [2026-10-20 04:06:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6033 [2026-10-20 04:06:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5746 [2026-10-20 04:06:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7177 [2026-10-20 04:06:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5192 [2026-10-20 04:06:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9236 [2026-10-20 04:06:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.2199 [2026-10-20 04:06:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.2420 [2026-10-20 04:06:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7475 [2026-10-20 04:06:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6003 [2026-10-20 04:06:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6363 [2026-10-20 04:06:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7664 [2026-10-20 04:07:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1191 [2026-10-20 04:07:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2098 [2026-10-20 04:07:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9925 [2026-10-20 04:07:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1803 [2026-10-20 04:07:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8462 [2026-10-20 04:07:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9738 [2026-10-20 04:07:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8172 [2026-10-20 04:07:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7809 [2026-10-20 04:07:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9376 [2026-10-20 04:07:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1538 [2026-10-20 04:07:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0679 [2026-10-20 04:07:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0376 [2026-10-20 04:07:59][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,572 | MFU: 1.8% [2026-10-20 04:08:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6425 [2026-10-20 04:08:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8001 [2026-10-20 04:08:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8476 [2026-10-20 04:08:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7725 [2026-10-20 04:08:04][INFO][training.pretrain_engine] [Ph2|Step 237] loss=7.0614 | ppl=1166.13 | aux=0.02511 | lr=1.68e-04 | gnorm=1.172 | tok/s=29,296 | MFU=7.0% | step=143172ms | tokens=0.256B | mem=4.2/32.7GB | cost=$124.41/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 23:56 (139.8h) | Phase3: SKIPPED | SFT: Oct 26 04:40 (144.5h) | DPO (Final): Oct 26 05:14 (145.1h) [2026-10-20 04:08:04][INFO][training.pretrain_engine] Starting global step 238 (accumulating 32 micro-batches)... [2026-10-20 04:08:04][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 04:08:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.4423 [2026-10-20 04:08:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.6793 [2026-10-20 04:08:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2475 [2026-10-20 04:08:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.3113 [2026-10-20 04:08:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1711 [2026-10-20 04:08:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8608 [2026-10-20 04:08:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0583 [2026-10-20 04:08:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4181 [2026-10-20 04:08:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4813 [2026-10-20 04:08:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9130 [2026-10-20 04:08:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1664 [2026-10-20 04:08:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0168 [2026-10-20 04:09:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6358 [2026-10-20 04:09:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9928 [2026-10-20 04:09:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6962 [2026-10-20 04:09:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9760 [2026-10-20 04:09:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.5643 [2026-10-20 04:09:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0333 [2026-10-20 04:09:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0805 [2026-10-20 04:09:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6986 [2026-10-20 04:09:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7670 [2026-10-20 04:09:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9751 [2026-10-20 04:09:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8980 [2026-10-20 04:09:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6307 [2026-10-20 04:10:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1950 [2026-10-20 04:10:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.5003 [2026-10-20 04:10:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3166 [2026-10-20 04:10:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.4519 [2026-10-20 04:10:23][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,532 | MFU: 1.8% [2026-10-20 04:10:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8144 [2026-10-20 04:10:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0328 [2026-10-20 04:10:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8488 [2026-10-20 04:10:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9822 [2026-10-20 04:10:28][INFO][training.pretrain_engine] [Ph2|Step 238] loss=7.0793 | ppl=1187.15 | aux=0.02516 | lr=1.69e-04 | gnorm=1.461 | tok/s=29,294 | MFU=7.0% | step=143180ms | tokens=0.257B | mem=4.2/32.7GB | cost=$124.94/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 25 23:58 (139.8h) | Phase3: SKIPPED | SFT: Oct 26 04:42 (144.5h) | DPO (Final): Oct 26 05:17 (145.1h) [2026-10-20 04:10:28][INFO][training.pretrain_engine] Starting global step 239 (accumulating 32 micro-batches)... [2026-10-20 04:10:28][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 04:10:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2111 [2026-10-20 04:10:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7618 [2026-10-20 04:10:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.8061 [2026-10-20 04:10:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1677 [2026-10-20 04:11:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1250 [2026-10-20 04:11:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8405 [2026-10-20 04:11:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0967 [2026-10-20 04:11:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.3574 [2026-10-20 04:11:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9298 [2026-10-20 04:11:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9032 [2026-10-20 04:11:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6239 [2026-10-20 04:11:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1998 [2026-10-20 04:11:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8222 [2026-10-20 04:11:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6634 [2026-10-20 04:11:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0634 [2026-10-20 04:11:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6138 [2026-10-20 04:11:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5841 [2026-10-20 04:11:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9605 [2026-10-20 04:11:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5917 [2026-10-20 04:11:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5106 [2026-10-20 04:12:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1347 [2026-10-20 04:12:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.8199 [2026-10-20 04:12:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9111 [2026-10-20 04:12:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0981 [2026-10-20 04:12:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2034 [2026-10-20 04:12:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.4084 [2026-10-20 04:12:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1036 [2026-10-20 04:12:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2218 [2026-10-20 04:12:46][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,558 | MFU: 1.8% [2026-10-20 04:12:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.1243 [2026-10-20 04:12:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3368 [2026-10-20 04:12:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8019 [2026-10-20 04:12:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2624 [2026-10-20 04:12:51][INFO][training.pretrain_engine] [Ph2|Step 239] loss=7.1302 | ppl=1249.07 | aux=0.02515 | lr=1.69e-04 | gnorm=0.848 | tok/s=29,294 | MFU=7.0% | step=143180ms | tokens=0.258B | mem=4.2/32.7GB | cost=$125.46/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 00:00 (139.8h) | Phase3: SKIPPED | SFT: Oct 26 04:44 (144.5h) | DPO (Final): Oct 26 05:18 (145.1h) [2026-10-20 04:12:51][INFO][training.pretrain_engine] Starting global step 240 (accumulating 32 micro-batches)... [2026-10-20 04:12:51][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 04:13:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8432 [2026-10-20 04:13:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0784 [2026-10-20 04:13:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1023 [2026-10-20 04:13:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9584 [2026-10-20 04:13:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1084 [2026-10-20 04:13:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7538 [2026-10-20 04:13:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0915 [2026-10-20 04:13:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1844 [2026-10-20 04:13:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.2475 [2026-10-20 04:13:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1802 [2026-10-20 04:13:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1198 [2026-10-20 04:13:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5539 [2026-10-20 04:14:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0772 [2026-10-20 04:14:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.3870 [2026-10-20 04:14:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.3879 [2026-10-20 04:14:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1477 [2026-10-20 04:14:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8682 [2026-10-20 04:14:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3293 [2026-10-20 04:14:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1106 [2026-10-20 04:14:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6884 [2026-10-20 04:14:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7701 [2026-10-20 04:14:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9136 [2026-10-20 04:14:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9177 [2026-10-20 04:14:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7122 [2026-10-20 04:14:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0077 [2026-10-20 04:14:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8190 [2026-10-20 04:14:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5302 [2026-10-20 04:14:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5224 [2026-10-20 04:15:09][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,572 | MFU: 1.8% [2026-10-20 04:15:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8033 [2026-10-20 04:15:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8746 [2026-10-20 04:15:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9111 [2026-10-20 04:15:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2427 [2026-10-20 04:15:14][INFO][training.pretrain_engine] [Ph2|Step 240] loss=6.9947 | ppl=1090.78 | aux=0.02514 | lr=1.70e-04 | gnorm=1.445 | tok/s=29,295 | MFU=7.0% | step=143174ms | tokens=0.259B | mem=4.2/32.7GB | cost=$125.99/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 00:01 (139.8h) | Phase3: SKIPPED | SFT: Oct 26 04:46 (144.5h) | DPO (Final): Oct 26 05:20 (145.1h) [2026-10-20 04:15:14][INFO][training.pretrain_engine] Starting global step 241 (accumulating 32 micro-batches)... [2026-10-20 04:15:14][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 04:15:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2816 [2026-10-20 04:15:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8247 [2026-10-20 04:15:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2498 [2026-10-20 04:15:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.3160 [2026-10-20 04:15:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0547 [2026-10-20 04:15:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0384 [2026-10-20 04:15:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1199 [2026-10-20 04:15:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.3084 [2026-10-20 04:16:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9802 [2026-10-20 04:16:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8630 [2026-10-20 04:16:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.2728 [2026-10-20 04:16:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7671 [2026-10-20 04:16:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1450 [2026-10-20 04:16:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9147 [2026-10-20 04:16:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9599 [2026-10-20 04:16:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0569 [2026-10-20 04:16:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1295 [2026-10-20 04:16:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3612 [2026-10-20 04:16:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3100 [2026-10-20 04:16:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2144 [2026-10-20 04:17:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0162 [2026-10-20 04:17:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0419 [2026-10-20 04:17:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8577 [2026-10-20 04:17:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.8180 [2026-10-20 04:17:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0634 [2026-10-20 04:17:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9123 [2026-10-20 04:17:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.7103 [2026-10-20 04:17:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9574 [2026-10-20 04:17:32][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,589 | MFU: 1.8% [2026-10-20 04:17:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4979 [2026-10-20 04:17:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0003 [2026-10-20 04:17:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.6468 [2026-10-20 04:17:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7071 [2026-10-20 04:17:37][INFO][training.pretrain_engine] [Ph2|Step 241] loss=7.1060 | ppl=1219.25 | aux=0.02518 | lr=1.70e-04 | gnorm=0.980 | tok/s=29,296 | MFU=7.0% | step=143170ms | tokens=0.260B | mem=4.2/32.7GB | cost=$126.51/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 00:03 (139.8h) | Phase3: SKIPPED | SFT: Oct 26 04:47 (144.5h) | DPO (Final): Oct 26 05:21 (145.1h) [2026-10-20 04:17:37][INFO][training.pretrain_engine] Starting global step 242 (accumulating 32 micro-batches)... [2026-10-20 04:17:37][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 04:17:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9225 [2026-10-20 04:17:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2137 [2026-10-20 04:17:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0977 [2026-10-20 04:17:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8468 [2026-10-20 04:18:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.7414 [2026-10-20 04:18:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.3018 [2026-10-20 04:18:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8893 [2026-10-20 04:18:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8705 [2026-10-20 04:18:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.2382 [2026-10-20 04:18:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.2523 [2026-10-20 04:18:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9573 [2026-10-20 04:18:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3055 [2026-10-20 04:18:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8945 [2026-10-20 04:18:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5315 [2026-10-20 04:18:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1693 [2026-10-20 04:18:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8001 [2026-10-20 04:19:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2517 [2026-10-20 04:19:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.5503 [2026-10-20 04:19:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2078 [2026-10-20 04:19:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3819 [2026-10-20 04:19:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2171 [2026-10-20 04:19:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.5593 [2026-10-20 04:19:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.3766 [2026-10-20 04:19:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.7815 [2026-10-20 04:19:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7671 [2026-10-20 04:19:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8714 [2026-10-20 04:19:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7704 [2026-10-20 04:19:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7222 [2026-10-20 04:19:55][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,577 | MFU: 1.8% [2026-10-20 04:20:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0237 [2026-10-20 04:20:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2009 [2026-10-20 04:20:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.6503 [2026-10-20 04:20:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.7048 [2026-10-20 04:20:00][INFO][training.pretrain_engine] [Ph2|Step 242] loss=7.0798 | ppl=1187.77 | aux=0.02514 | lr=1.71e-04 | gnorm=1.359 | tok/s=29,298 | MFU=7.0% | step=143160ms | tokens=0.261B | mem=4.2/32.7GB | cost=$127.03/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 00:04 (139.7h) | Phase3: SKIPPED | SFT: Oct 26 04:48 (144.5h) | DPO (Final): Oct 26 05:23 (145.1h) [2026-10-20 04:20:00][INFO][training.pretrain_engine] Starting global step 243 (accumulating 32 micro-batches)... [2026-10-20 04:20:00][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 04:20:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.3271 [2026-10-20 04:20:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8326 [2026-10-20 04:20:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4595 [2026-10-20 04:20:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4665 [2026-10-20 04:20:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6775 [2026-10-20 04:20:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0166 [2026-10-20 04:20:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6379 [2026-10-20 04:20:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0675 [2026-10-20 04:20:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3187 [2026-10-20 04:20:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9014 [2026-10-20 04:20:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7808 [2026-10-20 04:20:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0535 [2026-10-20 04:21:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6889 [2026-10-20 04:21:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6849 [2026-10-20 04:21:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8261 [2026-10-20 04:21:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9720 [2026-10-20 04:21:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9897 [2026-10-20 04:21:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9955 [2026-10-20 04:21:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1935 [2026-10-20 04:21:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0226 [2026-10-20 04:21:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7199 [2026-10-20 04:21:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.3274 [2026-10-20 04:21:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2400 [2026-10-20 04:21:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1241 [2026-10-20 04:22:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9210 [2026-10-20 04:22:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6522 [2026-10-20 04:22:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8194 [2026-10-20 04:22:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6976 [2026-10-20 04:22:18][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,576 | MFU: 1.8% [2026-10-20 04:22:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2418 [2026-10-20 04:22:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.5957 [2026-10-20 04:22:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4034 [2026-10-20 04:22:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3412 [2026-10-20 04:22:24][INFO][training.pretrain_engine] [Ph2|Step 243] loss=7.0252 | ppl=1124.63 | aux=0.02512 | lr=1.72e-04 | gnorm=0.801 | tok/s=29,298 | MFU=7.0% | step=143162ms | tokens=0.262B | mem=4.2/32.7GB | cost=$127.56/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 00:06 (139.7h) | Phase3: SKIPPED | SFT: Oct 26 04:50 (144.5h) | DPO (Final): Oct 26 05:24 (145.0h) [2026-10-20 04:22:27][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step243 | Tokens: 0.262B | Spent: $127.57 | Remaining: $272.43 [2026-10-20 04:22:27][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step243 [2026-10-20 04:22:27][INFO][training.pretrain_engine] Starting global step 244 (accumulating 32 micro-batches)... [2026-10-20 04:22:27][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 04:22:27][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 04:22:27][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 04:22:27][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 04:22:27][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step243 to Phase-Technologies/forge-500m-pretrain (path: phase2_step243)... [2026-10-20 04:22:27][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 04:22:37][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 04:22:37][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step243 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 04:22:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1864 [2026-10-20 04:22:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.4845 [2026-10-20 04:22:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6562 [2026-10-20 04:22:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.5739 [2026-10-20 04:22:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1343 [2026-10-20 04:22:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1082 [2026-10-20 04:22:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7715 [2026-10-20 04:23:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0082 [2026-10-20 04:23:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1392 [2026-10-20 04:23:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.7463 [2026-10-20 04:23:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.9410 [2026-10-20 04:23:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3039 [2026-10-20 04:23:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.7447 [2026-10-20 04:23:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2888 [2026-10-20 04:23:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4733 [2026-10-20 04:23:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4660 [2026-10-20 04:23:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0799 [2026-10-20 04:23:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.7796 [2026-10-20 04:23:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3867 [2026-10-20 04:23:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.9125 [2026-10-20 04:24:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2214 [2026-10-20 04:24:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2816 [2026-10-20 04:24:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9925 [2026-10-20 04:24:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6695 [2026-10-20 04:24:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3381 [2026-10-20 04:24:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.5629 [2026-10-20 04:24:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.5402 [2026-10-20 04:24:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9746 [2026-10-20 04:24:46][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,530 | MFU: 1.8% [2026-10-20 04:24:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7123 [2026-10-20 04:24:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.9074 [2026-10-20 04:24:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2764 [2026-10-20 04:24:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6501 [2026-10-20 04:24:51][INFO][training.pretrain_engine] [Ph2|Step 244] loss=7.0484 | ppl=1151.05 | aux=0.02513 | lr=1.72e-04 | gnorm=1.586 | tok/s=29,296 | MFU=7.0% | step=143169ms | tokens=0.263B | mem=4.2/32.7GB | cost=$128.09/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 00:08 (139.7h) | Phase3: SKIPPED | SFT: Oct 26 04:53 (144.5h) | DPO (Final): Oct 26 05:27 (145.0h) [2026-10-20 04:24:51][INFO][training.pretrain_engine] Starting global step 245 (accumulating 32 micro-batches)... [2026-10-20 04:24:51][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 04:25:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9848 [2026-10-20 04:25:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9699 [2026-10-20 04:25:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0574 [2026-10-20 04:25:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8591 [2026-10-20 04:25:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4049 [2026-10-20 04:25:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7549 [2026-10-20 04:25:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6457 [2026-10-20 04:25:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0715 [2026-10-20 04:25:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8668 [2026-10-20 04:25:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6816 [2026-10-20 04:25:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.2512 [2026-10-20 04:25:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7788 [2026-10-20 04:26:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8712 [2026-10-20 04:26:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9848 [2026-10-20 04:26:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0168 [2026-10-20 04:26:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1172 [2026-10-20 04:26:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5246 [2026-10-20 04:26:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7270 [2026-10-20 04:26:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9260 [2026-10-20 04:26:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2542 [2026-10-20 04:26:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6047 [2026-10-20 04:26:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6273 [2026-10-20 04:26:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8928 [2026-10-20 04:26:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6946 [2026-10-20 04:26:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2664 [2026-10-20 04:26:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.5035 [2026-10-20 04:26:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1991 [2026-10-20 04:26:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2922 [2026-10-20 04:27:10][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,537 | MFU: 1.8% [2026-10-20 04:27:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8828 [2026-10-20 04:27:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8565 [2026-10-20 04:27:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3277 [2026-10-20 04:27:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7600 [2026-10-20 04:27:14][INFO][training.pretrain_engine] [Ph2|Step 245] loss=6.9155 | ppl=1007.77 | aux=0.02512 | lr=1.73e-04 | gnorm=1.086 | tok/s=29,295 | MFU=7.0% | step=143173ms | tokens=0.264B | mem=4.2/32.7GB | cost=$128.62/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 00:10 (139.7h) | Phase3: SKIPPED | SFT: Oct 26 04:55 (144.5h) | DPO (Final): Oct 26 05:29 (145.0h) [2026-10-20 04:27:14][INFO][training.pretrain_engine] Starting global step 246 (accumulating 32 micro-batches)... [2026-10-20 04:27:14][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 04:27:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7728 [2026-10-20 04:27:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2266 [2026-10-20 04:27:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1525 [2026-10-20 04:27:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7572 [2026-10-20 04:27:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8018 [2026-10-20 04:27:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9264 [2026-10-20 04:27:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8907 [2026-10-20 04:27:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7365 [2026-10-20 04:28:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3257 [2026-10-20 04:28:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=8.2137 [2026-10-20 04:28:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0259 [2026-10-20 04:28:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3376 [2026-10-20 04:28:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6576 [2026-10-20 04:28:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8046 [2026-10-20 04:28:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7106 [2026-10-20 04:28:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0808 [2026-10-20 04:28:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4047 [2026-10-20 04:28:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5116 [2026-10-20 04:28:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0409 [2026-10-20 04:28:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.5955 [2026-10-20 04:29:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2872 [2026-10-20 04:29:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1892 [2026-10-20 04:29:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0952 [2026-10-20 04:29:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2058 [2026-10-20 04:29:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3254 [2026-10-20 04:29:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0597 [2026-10-20 04:29:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1527 [2026-10-20 04:29:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0171 [2026-10-20 04:29:33][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,549 | MFU: 1.8% [2026-10-20 04:29:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9857 [2026-10-20 04:29:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3943 [2026-10-20 04:29:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.1987 [2026-10-20 04:29:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2370 [2026-10-20 04:29:38][INFO][training.pretrain_engine] [Ph2|Step 246] loss=7.0591 | ppl=1163.37 | aux=0.02516 | lr=1.73e-04 | gnorm=2.172 | tok/s=29,294 | MFU=7.0% | step=143178ms | tokens=0.265B | mem=4.2/32.7GB | cost=$129.14/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 00:12 (139.7h) | Phase3: SKIPPED | SFT: Oct 26 04:57 (144.5h) | DPO (Final): Oct 26 05:31 (145.0h) [2026-10-20 04:29:38][INFO][training.pretrain_engine] Starting global step 247 (accumulating 32 micro-batches)... [2026-10-20 04:29:38][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 04:29:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5774 [2026-10-20 04:29:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9732 [2026-10-20 04:29:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5971 [2026-10-20 04:29:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2957 [2026-10-20 04:30:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1825 [2026-10-20 04:30:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2740 [2026-10-20 04:30:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.3634 [2026-10-20 04:30:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0650 [2026-10-20 04:30:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9152 [2026-10-20 04:30:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8914 [2026-10-20 04:30:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1200 [2026-10-20 04:30:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.2989 [2026-10-20 04:30:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6205 [2026-10-20 04:30:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0261 [2026-10-20 04:30:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0239 [2026-10-20 04:30:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0192 [2026-10-20 04:31:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9109 [2026-10-20 04:31:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9636 [2026-10-20 04:31:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3482 [2026-10-20 04:31:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1736 [2026-10-20 04:31:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7076 [2026-10-20 04:31:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9725 [2026-10-20 04:31:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6817 [2026-10-20 04:31:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5676 [2026-10-20 04:31:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5411 [2026-10-20 04:31:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8194 [2026-10-20 04:31:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6960 [2026-10-20 04:31:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7579 [2026-10-20 04:31:57][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,539 | MFU: 1.8% [2026-10-20 04:32:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7860 [2026-10-20 04:32:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0910 [2026-10-20 04:32:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9954 [2026-10-20 04:32:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3644 [2026-10-20 04:32:01][INFO][training.pretrain_engine] [Ph2|Step 247] loss=6.9715 | ppl=1065.78 | aux=0.02516 | lr=1.74e-04 | gnorm=1.617 | tok/s=29,294 | MFU=7.0% | step=143182ms | tokens=0.266B | mem=4.2/32.7GB | cost=$129.67/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 00:14 (139.7h) | Phase3: SKIPPED | SFT: Oct 26 04:59 (144.5h) | DPO (Final): Oct 26 05:33 (145.0h) [2026-10-20 04:32:01][INFO][training.pretrain_engine] Starting global step 248 (accumulating 32 micro-batches)... [2026-10-20 04:32:01][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 04:32:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1791 [2026-10-20 04:32:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1788 [2026-10-20 04:32:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1146 [2026-10-20 04:32:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0735 [2026-10-20 04:32:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0081 [2026-10-20 04:32:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9645 [2026-10-20 04:32:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8198 [2026-10-20 04:32:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2012 [2026-10-20 04:32:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1631 [2026-10-20 04:32:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1328 [2026-10-20 04:32:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0209 [2026-10-20 04:32:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0698 [2026-10-20 04:33:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6009 [2026-10-20 04:33:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6855 [2026-10-20 04:33:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3005 [2026-10-20 04:33:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4400 [2026-10-20 04:33:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8108 [2026-10-20 04:33:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5522 [2026-10-20 04:33:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9130 [2026-10-20 04:33:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6433 [2026-10-20 04:33:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9656 [2026-10-20 04:33:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9930 [2026-10-20 04:33:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0563 [2026-10-20 04:33:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8957 [2026-10-20 04:34:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5868 [2026-10-20 04:34:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1086 [2026-10-20 04:34:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0248 [2026-10-20 04:34:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8074 [2026-10-20 04:34:20][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,558 | MFU: 1.8% [2026-10-20 04:34:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3000 [2026-10-20 04:34:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4273 [2026-10-20 04:34:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8310 [2026-10-20 04:34:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0656 [2026-10-20 04:34:25][INFO][training.pretrain_engine] [Ph2|Step 248] loss=6.9946 | ppl=1090.78 | aux=0.02515 | lr=1.74e-04 | gnorm=1.695 | tok/s=29,293 | MFU=7.0% | step=143182ms | tokens=0.267B | mem=4.2/32.7GB | cost=$130.19/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 00:16 (139.7h) | Phase3: SKIPPED | SFT: Oct 26 05:01 (144.4h) | DPO (Final): Oct 26 05:35 (145.0h) [2026-10-20 04:34:25][INFO][training.pretrain_engine] Starting global step 249 (accumulating 32 micro-batches)... [2026-10-20 04:34:25][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 04:34:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2664 [2026-10-20 04:34:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8853 [2026-10-20 04:34:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1350 [2026-10-20 04:34:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6445 [2026-10-20 04:35:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7414 [2026-10-20 04:35:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4293 [2026-10-20 04:35:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9105 [2026-10-20 04:35:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6600 [2026-10-20 04:35:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3084 [2026-10-20 04:35:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7287 [2026-10-20 04:35:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8739 [2026-10-20 04:35:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9392 [2026-10-20 04:35:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1555 [2026-10-20 04:35:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.6723 [2026-10-20 04:35:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4886 [2026-10-20 04:35:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.5607 [2026-10-20 04:35:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5068 [2026-10-20 04:35:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8416 [2026-10-20 04:35:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7939 [2026-10-20 04:35:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8456 [2026-10-20 04:36:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1415 [2026-10-20 04:36:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0258 [2026-10-20 04:36:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9764 [2026-10-20 04:36:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8337 [2026-10-20 04:36:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2946 [2026-10-20 04:36:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2631 [2026-10-20 04:36:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3364 [2026-10-20 04:36:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0831 [2026-10-20 04:36:44][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,547 | MFU: 1.8% [2026-10-20 04:36:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9801 [2026-10-20 04:36:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6185 [2026-10-20 04:36:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7732 [2026-10-20 04:36:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.1095 [2026-10-20 04:36:48][INFO][training.pretrain_engine] [Ph2|Step 249] loss=6.8865 | ppl=978.95 | aux=0.02518 | lr=1.75e-04 | gnorm=1.570 | tok/s=29,293 | MFU=7.0% | step=143184ms | tokens=0.268B | mem=4.2/32.7GB | cost=$130.72/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 00:18 (139.7h) | Phase3: SKIPPED | SFT: Oct 26 05:02 (144.4h) | DPO (Final): Oct 26 05:37 (145.0h) [2026-10-20 04:36:48][INFO][training.pretrain_engine] Starting global step 250 (accumulating 32 micro-batches)... [2026-10-20 04:36:48][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 04:37:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7330 [2026-10-20 04:37:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6914 [2026-10-20 04:37:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6678 [2026-10-20 04:37:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6901 [2026-10-20 04:37:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6055 [2026-10-20 04:37:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9018 [2026-10-20 04:37:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9443 [2026-10-20 04:37:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8190 [2026-10-20 04:37:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5691 [2026-10-20 04:37:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8060 [2026-10-20 04:37:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7796 [2026-10-20 04:37:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8427 [2026-10-20 04:38:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.3053 [2026-10-20 04:38:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0690 [2026-10-20 04:38:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8944 [2026-10-20 04:38:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2227 [2026-10-20 04:38:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0195 [2026-10-20 04:38:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0122 [2026-10-20 04:38:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9468 [2026-10-20 04:38:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8078 [2026-10-20 04:38:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1269 [2026-10-20 04:38:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9979 [2026-10-20 04:38:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8509 [2026-10-20 04:38:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9486 [2026-10-20 04:38:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3039 [2026-10-20 04:38:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9519 [2026-10-20 04:38:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8307 [2026-10-20 04:38:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3893 [2026-10-20 04:39:07][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,580 | MFU: 1.8% [2026-10-20 04:39:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3393 [2026-10-20 04:39:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.5314 [2026-10-20 04:39:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6402 [2026-10-20 04:39:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.6231 [2026-10-20 04:39:11][INFO][training.pretrain_engine] [Ph2|Step 250] loss=6.8922 | ppl=984.61 | aux=0.02513 | lr=1.76e-04 | gnorm=1.023 | tok/s=29,294 | MFU=7.0% | step=143178ms | tokens=0.269B | mem=4.2/32.7GB | cost=$131.24/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 00:19 (139.7h) | Phase3: SKIPPED | SFT: Oct 26 05:04 (144.4h) | DPO (Final): Oct 26 05:38 (145.0h) [2026-10-20 04:39:11][INFO][training.pretrain_engine] Starting global step 251 (accumulating 32 micro-batches)... [2026-10-20 04:39:11][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 04:39:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.6680 [2026-10-20 04:39:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1153 [2026-10-20 04:39:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.3844 [2026-10-20 04:39:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9592 [2026-10-20 04:39:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4894 [2026-10-20 04:39:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4124 [2026-10-20 04:39:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.6088 [2026-10-20 04:39:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.3528 [2026-10-20 04:40:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7180 [2026-10-20 04:40:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7870 [2026-10-20 04:40:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7007 [2026-10-20 04:40:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9368 [2026-10-20 04:40:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2746 [2026-10-20 04:40:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5988 [2026-10-20 04:40:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8189 [2026-10-20 04:40:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7186 [2026-10-20 04:40:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7092 [2026-10-20 04:40:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2185 [2026-10-20 04:40:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.4894 [2026-10-20 04:40:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9600 [2026-10-20 04:40:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8699 [2026-10-20 04:40:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6336 [2026-10-20 04:40:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6168 [2026-10-20 04:40:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9245 [2026-10-20 04:41:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2789 [2026-10-20 04:41:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0250 [2026-10-20 04:41:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8906 [2026-10-20 04:41:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0263 [2026-10-20 04:41:30][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,556 | MFU: 1.8% [2026-10-20 04:41:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.8947 [2026-10-20 04:41:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4524 [2026-10-20 04:41:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3801 [2026-10-20 04:41:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.5017 [2026-10-20 04:41:34][INFO][training.pretrain_engine] [Ph2|Step 251] loss=7.0300 | ppl=1129.98 | aux=0.02519 | lr=1.76e-04 | gnorm=1.445 | tok/s=29,295 | MFU=7.0% | step=143177ms | tokens=0.271B | mem=4.2/32.7GB | cost=$131.76/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 00:21 (139.7h) | Phase3: SKIPPED | SFT: Oct 26 05:06 (144.4h) | DPO (Final): Oct 26 05:40 (145.0h) [2026-10-20 04:41:34][INFO][training.pretrain_engine] Starting global step 252 (accumulating 32 micro-batches)... [2026-10-20 04:41:34][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 04:41:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8769 [2026-10-20 04:41:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7437 [2026-10-20 04:41:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6367 [2026-10-20 04:41:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1534 [2026-10-20 04:42:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2275 [2026-10-20 04:42:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0270 [2026-10-20 04:42:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0196 [2026-10-20 04:42:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1216 [2026-10-20 04:42:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7755 [2026-10-20 04:42:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0640 [2026-10-20 04:42:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7956 [2026-10-20 04:42:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3242 [2026-10-20 04:42:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1268 [2026-10-20 04:42:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1348 [2026-10-20 04:42:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2281 [2026-10-20 04:42:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.6823 [2026-10-20 04:43:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7731 [2026-10-20 04:43:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8446 [2026-10-20 04:43:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8733 [2026-10-20 04:43:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9621 [2026-10-20 04:43:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0480 [2026-10-20 04:43:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1259 [2026-10-20 04:43:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2269 [2026-10-20 04:43:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7942 [2026-10-20 04:43:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.7760 [2026-10-20 04:43:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.8851 [2026-10-20 04:43:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.4803 [2026-10-20 04:43:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=8.0270 [2026-10-20 04:43:53][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,580 | MFU: 1.8% [2026-10-20 04:43:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8681 [2026-10-20 04:43:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6861 [2026-10-20 04:43:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9863 [2026-10-20 04:43:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0918 [2026-10-20 04:43:58][INFO][training.pretrain_engine] [Ph2|Step 252] loss=6.9910 | ppl=1086.85 | aux=0.02514 | lr=1.77e-04 | gnorm=1.406 | tok/s=29,295 | MFU=7.0% | step=143177ms | tokens=0.272B | mem=4.2/32.7GB | cost=$132.29/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 00:23 (139.7h) | Phase3: SKIPPED | SFT: Oct 26 05:07 (144.4h) | DPO (Final): Oct 26 05:42 (145.0h) [2026-10-20 04:44:01][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step252 | Tokens: 0.272B | Spent: $132.30 | Remaining: $267.70 [2026-10-20 04:44:01][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step252 [2026-10-20 04:44:01][INFO][training.pretrain_engine] Starting global step 253 (accumulating 32 micro-batches)... [2026-10-20 04:44:01][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 04:44:01][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 04:44:01][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 04:44:01][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 04:44:01][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step252 to Phase-Technologies/forge-500m-pretrain (path: phase2_step252)... [2026-10-20 04:44:01][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 04:44:10][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 04:44:10][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step252 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 04:44:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1836 [2026-10-20 04:44:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1756 [2026-10-20 04:44:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9582 [2026-10-20 04:44:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0849 [2026-10-20 04:44:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1385 [2026-10-20 04:44:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6826 [2026-10-20 04:44:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7766 [2026-10-20 04:44:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6998 [2026-10-20 04:44:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4318 [2026-10-20 04:44:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1720 [2026-10-20 04:44:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4955 [2026-10-20 04:44:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3656 [2026-10-20 04:45:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0161 [2026-10-20 04:45:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.3919 [2026-10-20 04:45:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8705 [2026-10-20 04:45:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2465 [2026-10-20 04:45:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.5081 [2026-10-20 04:45:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.4166 [2026-10-20 04:45:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2052 [2026-10-20 04:45:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3739 [2026-10-20 04:45:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6958 [2026-10-20 04:45:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.4337 [2026-10-20 04:45:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1641 [2026-10-20 04:45:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8463 [2026-10-20 04:46:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3634 [2026-10-20 04:46:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.5588 [2026-10-20 04:46:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1804 [2026-10-20 04:46:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9919 [2026-10-20 04:46:19][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,602 | MFU: 1.8% [2026-10-20 04:46:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9452 [2026-10-20 04:46:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8902 [2026-10-20 04:46:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9715 [2026-10-20 04:46:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9204 [2026-10-20 04:46:23][INFO][training.pretrain_engine] [Ph2|Step 253] loss=7.0959 | ppl=1206.98 | aux=0.02519 | lr=1.77e-04 | gnorm=1.320 | tok/s=29,297 | MFU=7.0% | step=143167ms | tokens=0.273B | mem=4.2/32.7GB | cost=$132.82/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 00:24 (139.6h) | Phase3: SKIPPED | SFT: Oct 26 05:09 (144.4h) | DPO (Final): Oct 26 05:43 (144.9h) [2026-10-20 04:46:23][INFO][training.pretrain_engine] Starting global step 254 (accumulating 32 micro-batches)... [2026-10-20 04:46:23][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 04:46:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.3983 [2026-10-20 04:46:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1615 [2026-10-20 04:46:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.3782 [2026-10-20 04:46:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0079 [2026-10-20 04:46:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6416 [2026-10-20 04:46:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8860 [2026-10-20 04:46:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7917 [2026-10-20 04:46:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6428 [2026-10-20 04:47:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5022 [2026-10-20 04:47:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8965 [2026-10-20 04:47:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7752 [2026-10-20 04:47:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7526 [2026-10-20 04:47:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0571 [2026-10-20 04:47:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7599 [2026-10-20 04:47:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9296 [2026-10-20 04:47:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1794 [2026-10-20 04:47:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9059 [2026-10-20 04:47:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9580 [2026-10-20 04:47:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8966 [2026-10-20 04:47:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9304 [2026-10-20 04:48:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2508 [2026-10-20 04:48:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5631 [2026-10-20 04:48:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6807 [2026-10-20 04:48:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8294 [2026-10-20 04:48:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7607 [2026-10-20 04:48:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7531 [2026-10-20 04:48:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7551 [2026-10-20 04:48:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5390 [2026-10-20 04:48:41][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,600 | MFU: 1.8% [2026-10-20 04:48:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0071 [2026-10-20 04:48:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7457 [2026-10-20 04:48:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7670 [2026-10-20 04:48:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8032 [2026-10-20 04:48:46][INFO][training.pretrain_engine] [Ph2|Step 254] loss=6.9812 | ppl=1076.22 | aux=0.02516 | lr=1.78e-04 | gnorm=2.219 | tok/s=29,298 | MFU=7.0% | step=143162ms | tokens=0.274B | mem=4.2/32.7GB | cost=$133.34/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 00:26 (139.6h) | Phase3: SKIPPED | SFT: Oct 26 05:10 (144.4h) | DPO (Final): Oct 26 05:44 (144.9h) [2026-10-20 04:48:46][INFO][training.pretrain_engine] Starting global step 255 (accumulating 32 micro-batches)... [2026-10-20 04:48:46][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 04:49:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5425 [2026-10-20 04:49:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0128 [2026-10-20 04:49:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9276 [2026-10-20 04:49:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9754 [2026-10-20 04:49:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6065 [2026-10-20 04:49:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9090 [2026-10-20 04:49:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0752 [2026-10-20 04:49:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1316 [2026-10-20 04:49:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.2019 [2026-10-20 04:49:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0854 [2026-10-20 04:49:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9273 [2026-10-20 04:49:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6922 [2026-10-20 04:49:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8239 [2026-10-20 04:49:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8424 [2026-10-20 04:49:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9245 [2026-10-20 04:49:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5920 [2026-10-20 04:50:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.7859 [2026-10-20 04:50:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9067 [2026-10-20 04:50:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9255 [2026-10-20 04:50:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9544 [2026-10-20 04:50:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8080 [2026-10-20 04:50:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6185 [2026-10-20 04:50:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8859 [2026-10-20 04:50:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6524 [2026-10-20 04:50:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0294 [2026-10-20 04:50:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7134 [2026-10-20 04:50:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6426 [2026-10-20 04:50:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9521 [2026-10-20 04:51:04][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,590 | MFU: 1.8% [2026-10-20 04:51:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7427 [2026-10-20 04:51:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0336 [2026-10-20 04:51:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3409 [2026-10-20 04:51:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7894 [2026-10-20 04:51:10][INFO][training.pretrain_engine] [Ph2|Step 255] loss=6.9805 | ppl=1075.42 | aux=0.02517 | lr=1.78e-04 | gnorm=1.469 | tok/s=29,298 | MFU=7.0% | step=143159ms | tokens=0.275B | mem=4.2/32.7GB | cost=$133.87/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 00:27 (139.6h) | Phase3: SKIPPED | SFT: Oct 26 05:12 (144.4h) | DPO (Final): Oct 26 05:46 (144.9h) [2026-10-20 04:51:10][INFO][training.pretrain_engine] Starting global step 256 (accumulating 32 micro-batches)... [2026-10-20 04:51:10][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 04:51:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.4365 [2026-10-20 04:51:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.3560 [2026-10-20 04:51:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.3338 [2026-10-20 04:51:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.3655 [2026-10-20 04:51:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4113 [2026-10-20 04:51:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7625 [2026-10-20 04:51:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4993 [2026-10-20 04:51:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8176 [2026-10-20 04:52:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5646 [2026-10-20 04:52:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4844 [2026-10-20 04:52:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6101 [2026-10-20 04:52:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5182 [2026-10-20 04:52:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8829 [2026-10-20 04:52:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0295 [2026-10-20 04:52:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8136 [2026-10-20 04:52:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6045 [2026-10-20 04:52:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.5465 [2026-10-20 04:52:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3354 [2026-10-20 04:52:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3451 [2026-10-20 04:52:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.4870 [2026-10-20 04:52:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0080 [2026-10-20 04:52:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=8.0268 [2026-10-20 04:52:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2005 [2026-10-20 04:52:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9008 [2026-10-20 04:53:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9915 [2026-10-20 04:53:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7750 [2026-10-20 04:53:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0093 [2026-10-20 04:53:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7240 [2026-10-20 04:53:27][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,620 | MFU: 1.8% [2026-10-20 04:53:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8505 [2026-10-20 04:53:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7284 [2026-10-20 04:53:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7914 [2026-10-20 04:53:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5886 [2026-10-20 04:53:32][INFO][training.pretrain_engine] [Ph2|Step 256] loss=6.9786 | ppl=1073.41 | aux=0.02517 | lr=1.79e-04 | gnorm=1.758 | tok/s=29,299 | MFU=7.0% | step=143153ms | tokens=0.276B | mem=4.2/32.7GB | cost=$134.39/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 00:29 (139.6h) | Phase3: SKIPPED | SFT: Oct 26 05:13 (144.3h) | DPO (Final): Oct 26 05:47 (144.9h) [2026-10-20 04:53:32][INFO][training.pretrain_engine] Starting global step 257 (accumulating 32 micro-batches)... [2026-10-20 04:53:32][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 04:53:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9848 [2026-10-20 04:53:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0635 [2026-10-20 04:53:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8543 [2026-10-20 04:53:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8746 [2026-10-20 04:54:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7238 [2026-10-20 04:54:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2447 [2026-10-20 04:54:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4385 [2026-10-20 04:54:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1391 [2026-10-20 04:54:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.2627 [2026-10-20 04:54:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9153 [2026-10-20 04:54:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1753 [2026-10-20 04:54:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9767 [2026-10-20 04:54:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.7965 [2026-10-20 04:54:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.7322 [2026-10-20 04:54:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.7569 [2026-10-20 04:54:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0353 [2026-10-20 04:55:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7869 [2026-10-20 04:55:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6132 [2026-10-20 04:55:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6938 [2026-10-20 04:55:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8327 [2026-10-20 04:55:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7525 [2026-10-20 04:55:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8755 [2026-10-20 04:55:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8061 [2026-10-20 04:55:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8900 [2026-10-20 04:55:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5627 [2026-10-20 04:55:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2578 [2026-10-20 04:55:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.7909 [2026-10-20 04:55:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9804 [2026-10-20 04:55:51][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,595 | MFU: 1.8% [2026-10-20 04:55:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.5516 [2026-10-20 04:55:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9512 [2026-10-20 04:55:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.6483 [2026-10-20 04:55:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.5741 [2026-10-20 04:55:55][INFO][training.pretrain_engine] [Ph2|Step 257] loss=6.9266 | ppl=1019.01 | aux=0.02513 | lr=1.79e-04 | gnorm=0.930 | tok/s=29,300 | MFU=7.0% | step=143148ms | tokens=0.277B | mem=4.2/32.7GB | cost=$134.91/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 00:30 (139.6h) | Phase3: SKIPPED | SFT: Oct 26 05:15 (144.3h) | DPO (Final): Oct 26 05:49 (144.9h) [2026-10-20 04:55:55][INFO][training.pretrain_engine] Starting global step 258 (accumulating 32 micro-batches)... [2026-10-20 04:55:55][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 04:56:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8964 [2026-10-20 04:56:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9421 [2026-10-20 04:56:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8648 [2026-10-20 04:56:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9491 [2026-10-20 04:56:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7572 [2026-10-20 04:56:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6570 [2026-10-20 04:56:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7446 [2026-10-20 04:56:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1900 [2026-10-20 04:56:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1985 [2026-10-20 04:56:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6703 [2026-10-20 04:56:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7822 [2026-10-20 04:56:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8482 [2026-10-20 04:57:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2978 [2026-10-20 04:57:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4136 [2026-10-20 04:57:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8912 [2026-10-20 04:57:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0518 [2026-10-20 04:57:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8129 [2026-10-20 04:57:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4758 [2026-10-20 04:57:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6342 [2026-10-20 04:57:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9850 [2026-10-20 04:57:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0376 [2026-10-20 04:57:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9239 [2026-10-20 04:57:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.9512 [2026-10-20 04:57:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9593 [2026-10-20 04:58:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9045 [2026-10-20 04:58:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5495 [2026-10-20 04:58:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7595 [2026-10-20 04:58:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6348 [2026-10-20 04:58:14][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,574 | MFU: 1.8% [2026-10-20 04:58:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1603 [2026-10-20 04:58:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5541 [2026-10-20 04:58:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.1240 [2026-10-20 04:58:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7264 [2026-10-20 04:58:18][INFO][training.pretrain_engine] [Ph2|Step 258] loss=6.9981 | ppl=1094.57 | aux=0.02516 | lr=1.80e-04 | gnorm=1.211 | tok/s=29,302 | MFU=7.0% | step=143141ms | tokens=0.278B | mem=4.2/32.7GB | cost=$135.43/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 00:32 (139.6h) | Phase3: SKIPPED | SFT: Oct 26 05:16 (144.3h) | DPO (Final): Oct 26 05:50 (144.9h) [2026-10-20 04:58:18][INFO][training.pretrain_engine] Starting global step 259 (accumulating 32 micro-batches)... [2026-10-20 04:58:18][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 04:58:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8370 [2026-10-20 04:58:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7075 [2026-10-20 04:58:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.4701 [2026-10-20 04:58:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8112 [2026-10-20 04:58:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2688 [2026-10-20 04:58:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.3455 [2026-10-20 04:58:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2454 [2026-10-20 04:58:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9495 [2026-10-20 04:59:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8244 [2026-10-20 04:59:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8151 [2026-10-20 04:59:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8567 [2026-10-20 04:59:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9139 [2026-10-20 04:59:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7611 [2026-10-20 04:59:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7553 [2026-10-20 04:59:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7492 [2026-10-20 04:59:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7405 [2026-10-20 04:59:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0963 [2026-10-20 04:59:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8662 [2026-10-20 04:59:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7461 [2026-10-20 04:59:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7822 [2026-10-20 05:00:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8796 [2026-10-20 05:00:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7341 [2026-10-20 05:00:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7769 [2026-10-20 05:00:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5991 [2026-10-20 05:00:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9940 [2026-10-20 05:00:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4979 [2026-10-20 05:00:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1880 [2026-10-20 05:00:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7140 [2026-10-20 05:00:36][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,600 | MFU: 1.8% [2026-10-20 05:00:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.1912 [2026-10-20 05:00:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.8096 [2026-10-20 05:00:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0421 [2026-10-20 05:00:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.5259 [2026-10-20 05:00:41][INFO][training.pretrain_engine] [Ph2|Step 259] loss=6.9341 | ppl=1026.66 | aux=0.02514 | lr=1.81e-04 | gnorm=0.953 | tok/s=29,304 | MFU=7.0% | step=143130ms | tokens=0.279B | mem=4.2/32.7GB | cost=$135.95/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 00:33 (139.5h) | Phase3: SKIPPED | SFT: Oct 26 05:17 (144.3h) | DPO (Final): Oct 26 05:51 (144.9h) [2026-10-20 05:00:41][INFO][training.pretrain_engine] Starting global step 260 (accumulating 32 micro-batches)... [2026-10-20 05:00:41][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 05:00:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.3436 [2026-10-20 05:00:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9676 [2026-10-20 05:00:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0918 [2026-10-20 05:00:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1756 [2026-10-20 05:01:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2431 [2026-10-20 05:01:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8901 [2026-10-20 05:01:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0722 [2026-10-20 05:01:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1735 [2026-10-20 05:01:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9886 [2026-10-20 05:01:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1589 [2026-10-20 05:01:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9072 [2026-10-20 05:01:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9355 [2026-10-20 05:01:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3795 [2026-10-20 05:01:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5823 [2026-10-20 05:01:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6150 [2026-10-20 05:01:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6701 [2026-10-20 05:02:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8029 [2026-10-20 05:02:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9923 [2026-10-20 05:02:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0007 [2026-10-20 05:02:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8518 [2026-10-20 05:02:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7322 [2026-10-20 05:02:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5917 [2026-10-20 05:02:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5009 [2026-10-20 05:02:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9144 [2026-10-20 05:02:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6132 [2026-10-20 05:02:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4967 [2026-10-20 05:02:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5449 [2026-10-20 05:02:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5410 [2026-10-20 05:02:59][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,585 | MFU: 1.8% [2026-10-20 05:03:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4703 [2026-10-20 05:03:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8286 [2026-10-20 05:03:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6949 [2026-10-20 05:03:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9035 [2026-10-20 05:03:04][INFO][training.pretrain_engine] [Ph2|Step 260] loss=6.9494 | ppl=1042.51 | aux=0.02513 | lr=1.81e-04 | gnorm=0.961 | tok/s=29,309 | MFU=7.0% | step=143108ms | tokens=0.280B | mem=4.2/32.7GB | cost=$136.48/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 00:33 (139.5h) | Phase3: SKIPPED | SFT: Oct 26 05:18 (144.3h) | DPO (Final): Oct 26 05:52 (144.8h) [2026-10-20 05:03:04][INFO][training.pretrain_engine] Starting global step 261 (accumulating 32 micro-batches)... [2026-10-20 05:03:04][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 05:03:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9076 [2026-10-20 05:03:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.3746 [2026-10-20 05:03:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.8409 [2026-10-20 05:03:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9076 [2026-10-20 05:03:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9987 [2026-10-20 05:03:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1290 [2026-10-20 05:03:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8745 [2026-10-20 05:03:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2530 [2026-10-20 05:03:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4339 [2026-10-20 05:03:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6009 [2026-10-20 05:03:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5601 [2026-10-20 05:03:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6085 [2026-10-20 05:04:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1651 [2026-10-20 05:04:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9679 [2026-10-20 05:04:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0806 [2026-10-20 05:04:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7955 [2026-10-20 05:04:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3021 [2026-10-20 05:04:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2689 [2026-10-20 05:04:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3835 [2026-10-20 05:04:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3856 [2026-10-20 05:04:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.5799 [2026-10-20 05:04:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6777 [2026-10-20 05:04:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.8288 [2026-10-20 05:04:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.4422 [2026-10-20 05:05:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.6467 [2026-10-20 05:05:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.6290 [2026-10-20 05:05:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3164 [2026-10-20 05:05:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.4465 [2026-10-20 05:05:22][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,578 | MFU: 1.8% [2026-10-20 05:05:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0739 [2026-10-20 05:05:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5850 [2026-10-20 05:05:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5810 [2026-10-20 05:05:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7795 [2026-10-20 05:05:26][INFO][training.pretrain_engine] [Ph2|Step 261] loss=6.9454 | ppl=1038.41 | aux=0.02515 | lr=1.82e-04 | gnorm=1.320 | tok/s=29,312 | MFU=7.0% | step=143091ms | tokens=0.281B | mem=4.2/32.7GB | cost=$137.00/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 00:34 (139.5h) | Phase3: SKIPPED | SFT: Oct 26 05:18 (144.2h) | DPO (Final): Oct 26 05:52 (144.8h) [2026-10-20 05:05:30][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step261 | Tokens: 0.281B | Spent: $137.01 | Remaining: $262.99 [2026-10-20 05:05:30][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step261 [2026-10-20 05:05:30][INFO][training.pretrain_engine] Starting global step 262 (accumulating 32 micro-batches)... [2026-10-20 05:05:30][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 05:05:30][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 05:05:30][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 05:05:30][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 05:05:30][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step261 to Phase-Technologies/forge-500m-pretrain (path: phase2_step261)... [2026-10-20 05:05:30][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 05:05:39][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 05:05:39][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step261 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 05:05:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8875 [2026-10-20 05:05:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6131 [2026-10-20 05:05:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6458 [2026-10-20 05:05:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8082 [2026-10-20 05:06:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6794 [2026-10-20 05:06:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.3892 [2026-10-20 05:06:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0137 [2026-10-20 05:06:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7890 [2026-10-20 05:06:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2655 [2026-10-20 05:06:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7658 [2026-10-20 05:06:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3404 [2026-10-20 05:06:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1359 [2026-10-20 05:06:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7178 [2026-10-20 05:06:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6114 [2026-10-20 05:06:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7687 [2026-10-20 05:06:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8090 [2026-10-20 05:06:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5948 [2026-10-20 05:06:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6527 [2026-10-20 05:06:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7823 [2026-10-20 05:06:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9189 [2026-10-20 05:07:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7301 [2026-10-20 05:07:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7465 [2026-10-20 05:07:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7542 [2026-10-20 05:07:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7975 [2026-10-20 05:07:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9183 [2026-10-20 05:07:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6092 [2026-10-20 05:07:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9580 [2026-10-20 05:07:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6237 [2026-10-20 05:07:49][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,563 | MFU: 1.8% [2026-10-20 05:07:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2156 [2026-10-20 05:07:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3021 [2026-10-20 05:07:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0098 [2026-10-20 05:07:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3491 [2026-10-20 05:07:53][INFO][training.pretrain_engine] [Ph2|Step 262] loss=6.9368 | ppl=1029.45 | aux=0.02515 | lr=1.82e-04 | gnorm=1.008 | tok/s=29,316 | MFU=7.0% | step=143074ms | tokens=0.282B | mem=4.2/32.7GB | cost=$137.53/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 00:35 (139.5h) | Phase3: SKIPPED | SFT: Oct 26 05:19 (144.2h) | DPO (Final): Oct 26 05:53 (144.8h) [2026-10-20 05:07:53][INFO][training.pretrain_engine] Starting global step 263 (accumulating 32 micro-batches)... [2026-10-20 05:07:53][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 05:08:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7889 [2026-10-20 05:08:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2931 [2026-10-20 05:08:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4721 [2026-10-20 05:08:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4929 [2026-10-20 05:08:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5818 [2026-10-20 05:08:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5549 [2026-10-20 05:08:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7136 [2026-10-20 05:08:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1545 [2026-10-20 05:08:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9305 [2026-10-20 05:08:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3890 [2026-10-20 05:08:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1432 [2026-10-20 05:08:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6465 [2026-10-20 05:09:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1462 [2026-10-20 05:09:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2120 [2026-10-20 05:09:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0557 [2026-10-20 05:09:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0176 [2026-10-20 05:09:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5952 [2026-10-20 05:09:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0375 [2026-10-20 05:09:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9006 [2026-10-20 05:09:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7665 [2026-10-20 05:09:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9246 [2026-10-20 05:09:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6955 [2026-10-20 05:09:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1334 [2026-10-20 05:09:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2533 [2026-10-20 05:09:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6142 [2026-10-20 05:09:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4132 [2026-10-20 05:09:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9257 [2026-10-20 05:09:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1470 [2026-10-20 05:10:11][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,597 | MFU: 1.8% [2026-10-20 05:10:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2084 [2026-10-20 05:10:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3501 [2026-10-20 05:10:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.1934 [2026-10-20 05:10:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4597 [2026-10-20 05:10:16][INFO][training.pretrain_engine] [Ph2|Step 263] loss=6.8710 | ppl=963.89 | aux=0.02512 | lr=1.83e-04 | gnorm=0.918 | tok/s=29,315 | MFU=7.0% | step=143076ms | tokens=0.283B | mem=4.2/32.7GB | cost=$138.06/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 00:37 (139.5h) | Phase3: SKIPPED | SFT: Oct 26 05:21 (144.2h) | DPO (Final): Oct 26 05:55 (144.8h) [2026-10-20 05:10:16][INFO][training.pretrain_engine] Starting global step 264 (accumulating 32 micro-batches)... [2026-10-20 05:10:16][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 05:10:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0152 [2026-10-20 05:10:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9980 [2026-10-20 05:10:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1311 [2026-10-20 05:10:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0689 [2026-10-20 05:10:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9656 [2026-10-20 05:10:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9452 [2026-10-20 05:10:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2365 [2026-10-20 05:10:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9236 [2026-10-20 05:11:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8443 [2026-10-20 05:11:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6817 [2026-10-20 05:11:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7215 [2026-10-20 05:11:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8477 [2026-10-20 05:11:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4030 [2026-10-20 05:11:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2647 [2026-10-20 05:11:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3570 [2026-10-20 05:11:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2761 [2026-10-20 05:11:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0594 [2026-10-20 05:11:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0569 [2026-10-20 05:11:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8842 [2026-10-20 05:11:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0729 [2026-10-20 05:12:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7959 [2026-10-20 05:12:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3689 [2026-10-20 05:12:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1795 [2026-10-20 05:12:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6289 [2026-10-20 05:12:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4907 [2026-10-20 05:12:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.5304 [2026-10-20 05:12:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0197 [2026-10-20 05:12:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2801 [2026-10-20 05:12:34][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,596 | MFU: 1.8% [2026-10-20 05:12:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2787 [2026-10-20 05:12:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9428 [2026-10-20 05:12:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8326 [2026-10-20 05:12:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2063 [2026-10-20 05:12:39][INFO][training.pretrain_engine] [Ph2|Step 264] loss=6.8476 | ppl=941.63 | aux=0.02513 | lr=1.83e-04 | gnorm=1.523 | tok/s=29,315 | MFU=7.0% | step=143075ms | tokens=0.284B | mem=4.2/32.7GB | cost=$138.58/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 00:39 (139.4h) | Phase3: SKIPPED | SFT: Oct 26 05:23 (144.2h) | DPO (Final): Oct 26 05:57 (144.7h) [2026-10-20 05:12:39][INFO][training.pretrain_engine] Starting global step 265 (accumulating 32 micro-batches)... [2026-10-20 05:12:39][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 05:12:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7772 [2026-10-20 05:12:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8251 [2026-10-20 05:12:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9470 [2026-10-20 05:12:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7591 [2026-10-20 05:13:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4490 [2026-10-20 05:13:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4880 [2026-10-20 05:13:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7038 [2026-10-20 05:13:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.3496 [2026-10-20 05:13:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8342 [2026-10-20 05:13:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0241 [2026-10-20 05:13:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7930 [2026-10-20 05:13:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0229 [2026-10-20 05:13:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6327 [2026-10-20 05:13:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9629 [2026-10-20 05:13:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6117 [2026-10-20 05:13:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8471 [2026-10-20 05:14:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9472 [2026-10-20 05:14:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1571 [2026-10-20 05:14:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7443 [2026-10-20 05:14:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8841 [2026-10-20 05:14:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7159 [2026-10-20 05:14:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1530 [2026-10-20 05:14:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9966 [2026-10-20 05:14:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8197 [2026-10-20 05:14:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0772 [2026-10-20 05:14:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.5216 [2026-10-20 05:14:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.6451 [2026-10-20 05:14:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9640 [2026-10-20 05:14:57][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,595 | MFU: 1.8% [2026-10-20 05:15:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.5415 [2026-10-20 05:15:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7465 [2026-10-20 05:15:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8149 [2026-10-20 05:15:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.1314 [2026-10-20 05:15:02][INFO][training.pretrain_engine] [Ph2|Step 265] loss=6.8882 | ppl=980.67 | aux=0.02518 | lr=1.84e-04 | gnorm=1.211 | tok/s=29,315 | MFU=7.0% | step=143075ms | tokens=0.285B | mem=4.2/32.7GB | cost=$139.10/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 00:40 (139.4h) | Phase3: SKIPPED | SFT: Oct 26 05:25 (144.2h) | DPO (Final): Oct 26 05:59 (144.7h) [2026-10-20 05:15:02][INFO][training.pretrain_engine] Starting global step 266 (accumulating 32 micro-batches)... [2026-10-20 05:15:02][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 05:15:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8419 [2026-10-20 05:15:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3556 [2026-10-20 05:15:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4473 [2026-10-20 05:15:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7649 [2026-10-20 05:15:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5125 [2026-10-20 05:15:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5075 [2026-10-20 05:15:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1832 [2026-10-20 05:15:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5544 [2026-10-20 05:15:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0126 [2026-10-20 05:15:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4799 [2026-10-20 05:15:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0553 [2026-10-20 05:15:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5626 [2026-10-20 05:16:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5928 [2026-10-20 05:16:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0654 [2026-10-20 05:16:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4850 [2026-10-20 05:16:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5849 [2026-10-20 05:16:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8551 [2026-10-20 05:16:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3544 [2026-10-20 05:16:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.6020 [2026-10-20 05:16:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.4901 [2026-10-20 05:16:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9443 [2026-10-20 05:16:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0734 [2026-10-20 05:16:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1285 [2026-10-20 05:16:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0053 [2026-10-20 05:17:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9234 [2026-10-20 05:17:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9035 [2026-10-20 05:17:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7750 [2026-10-20 05:17:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3296 [2026-10-20 05:17:20][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,568 | MFU: 1.8% [2026-10-20 05:17:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7536 [2026-10-20 05:17:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4983 [2026-10-20 05:17:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6394 [2026-10-20 05:17:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8958 [2026-10-20 05:17:25][INFO][training.pretrain_engine] [Ph2|Step 266] loss=6.8434 | ppl=937.65 | aux=0.02516 | lr=1.85e-04 | gnorm=1.328 | tok/s=29,315 | MFU=7.0% | step=143076ms | tokens=0.286B | mem=4.2/32.7GB | cost=$139.62/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 00:42 (139.4h) | Phase3: SKIPPED | SFT: Oct 26 05:26 (144.2h) | DPO (Final): Oct 26 06:01 (144.7h) [2026-10-20 05:17:25][INFO][training.pretrain_engine] Starting global step 267 (accumulating 32 micro-batches)... [2026-10-20 05:17:25][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 05:17:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2545 [2026-10-20 05:17:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1702 [2026-10-20 05:17:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2506 [2026-10-20 05:17:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2848 [2026-10-20 05:18:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2685 [2026-10-20 05:18:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9280 [2026-10-20 05:18:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.5732 [2026-10-20 05:18:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0436 [2026-10-20 05:18:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9344 [2026-10-20 05:18:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5245 [2026-10-20 05:18:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7630 [2026-10-20 05:18:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3683 [2026-10-20 05:18:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1288 [2026-10-20 05:18:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6253 [2026-10-20 05:18:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7891 [2026-10-20 05:18:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7895 [2026-10-20 05:18:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.9634 [2026-10-20 05:18:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0840 [2026-10-20 05:18:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.4522 [2026-10-20 05:18:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8609 [2026-10-20 05:19:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6563 [2026-10-20 05:19:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8776 [2026-10-20 05:19:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0322 [2026-10-20 05:19:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3315 [2026-10-20 05:19:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9934 [2026-10-20 05:19:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1253 [2026-10-20 05:19:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8202 [2026-10-20 05:19:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0484 [2026-10-20 05:19:44][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,563 | MFU: 1.8% [2026-10-20 05:19:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0209 [2026-10-20 05:19:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9074 [2026-10-20 05:19:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0810 [2026-10-20 05:19:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7339 [2026-10-20 05:19:48][INFO][training.pretrain_engine] [Ph2|Step 267] loss=6.8732 | ppl=966.01 | aux=0.02514 | lr=1.85e-04 | gnorm=1.008 | tok/s=29,315 | MFU=7.0% | step=143078ms | tokens=0.287B | mem=4.2/32.7GB | cost=$140.15/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 00:44 (139.4h) | Phase3: SKIPPED | SFT: Oct 26 05:28 (144.2h) | DPO (Final): Oct 26 06:02 (144.7h) [2026-10-20 05:19:48][INFO][training.pretrain_engine] Starting global step 268 (accumulating 32 micro-batches)... [2026-10-20 05:19:48][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 05:20:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9007 [2026-10-20 05:20:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0584 [2026-10-20 05:20:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7028 [2026-10-20 05:20:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8401 [2026-10-20 05:20:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4094 [2026-10-20 05:20:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1888 [2026-10-20 05:20:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2015 [2026-10-20 05:20:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6408 [2026-10-20 05:20:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2902 [2026-10-20 05:20:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9397 [2026-10-20 05:20:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6335 [2026-10-20 05:20:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8890 [2026-10-20 05:20:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1903 [2026-10-20 05:20:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1562 [2026-10-20 05:21:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9395 [2026-10-20 05:21:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4522 [2026-10-20 05:21:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7123 [2026-10-20 05:21:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7650 [2026-10-20 05:21:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6400 [2026-10-20 05:21:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5643 [2026-10-20 05:21:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9560 [2026-10-20 05:21:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9006 [2026-10-20 05:21:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0459 [2026-10-20 05:21:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.5251 [2026-10-20 05:21:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3521 [2026-10-20 05:21:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.9420 [2026-10-20 05:21:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.8131 [2026-10-20 05:21:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1720 [2026-10-20 05:22:07][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,568 | MFU: 1.8% [2026-10-20 05:22:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6744 [2026-10-20 05:22:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7796 [2026-10-20 05:22:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8928 [2026-10-20 05:22:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8280 [2026-10-20 05:22:11][INFO][training.pretrain_engine] [Ph2|Step 268] loss=6.9046 | ppl=996.85 | aux=0.02514 | lr=1.86e-04 | gnorm=1.133 | tok/s=29,314 | MFU=7.0% | step=143080ms | tokens=0.288B | mem=4.2/32.7GB | cost=$140.67/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 00:46 (139.4h) | Phase3: SKIPPED | SFT: Oct 26 05:30 (144.1h) | DPO (Final): Oct 26 06:04 (144.7h) [2026-10-20 05:22:11][INFO][training.pretrain_engine] Starting global step 269 (accumulating 32 micro-batches)... [2026-10-20 05:22:11][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 05:22:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6112 [2026-10-20 05:22:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1185 [2026-10-20 05:22:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.3448 [2026-10-20 05:22:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6582 [2026-10-20 05:22:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8971 [2026-10-20 05:22:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2082 [2026-10-20 05:22:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8890 [2026-10-20 05:22:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6967 [2026-10-20 05:23:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7300 [2026-10-20 05:23:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1826 [2026-10-20 05:23:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.6979 [2026-10-20 05:23:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8220 [2026-10-20 05:23:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1609 [2026-10-20 05:23:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7650 [2026-10-20 05:23:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8709 [2026-10-20 05:23:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.5112 [2026-10-20 05:23:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7383 [2026-10-20 05:23:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7994 [2026-10-20 05:23:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8574 [2026-10-20 05:23:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5385 [2026-10-20 05:23:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2604 [2026-10-20 05:23:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.4097 [2026-10-20 05:23:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2128 [2026-10-20 05:23:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.3486 [2026-10-20 05:24:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6627 [2026-10-20 05:24:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1092 [2026-10-20 05:24:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6311 [2026-10-20 05:24:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9929 [2026-10-20 05:24:30][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,579 | MFU: 1.8% [2026-10-20 05:24:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1562 [2026-10-20 05:24:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6478 [2026-10-20 05:24:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4423 [2026-10-20 05:24:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1929 [2026-10-20 05:24:34][INFO][training.pretrain_engine] [Ph2|Step 269] loss=6.8713 | ppl=964.20 | aux=0.02512 | lr=1.86e-04 | gnorm=1.148 | tok/s=29,315 | MFU=7.0% | step=143076ms | tokens=0.289B | mem=4.2/32.7GB | cost=$141.19/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 00:48 (139.4h) | Phase3: SKIPPED | SFT: Oct 26 05:32 (144.1h) | DPO (Final): Oct 26 06:06 (144.7h) [2026-10-20 05:24:34][INFO][training.pretrain_engine] Starting global step 270 (accumulating 32 micro-batches)... [2026-10-20 05:24:34][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 05:24:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.7402 [2026-10-20 05:24:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.6690 [2026-10-20 05:24:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9036 [2026-10-20 05:24:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2370 [2026-10-20 05:25:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9261 [2026-10-20 05:25:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6347 [2026-10-20 05:25:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7424 [2026-10-20 05:25:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6371 [2026-10-20 05:25:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6586 [2026-10-20 05:25:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7337 [2026-10-20 05:25:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3785 [2026-10-20 05:25:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7615 [2026-10-20 05:25:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5986 [2026-10-20 05:25:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6329 [2026-10-20 05:25:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5154 [2026-10-20 05:25:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7234 [2026-10-20 05:26:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2371 [2026-10-20 05:26:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9041 [2026-10-20 05:26:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1481 [2026-10-20 05:26:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1525 [2026-10-20 05:26:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4717 [2026-10-20 05:26:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.3413 [2026-10-20 05:26:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0297 [2026-10-20 05:26:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6993 [2026-10-20 05:26:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4845 [2026-10-20 05:26:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8915 [2026-10-20 05:26:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6302 [2026-10-20 05:26:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7879 [2026-10-20 05:26:52][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,581 | MFU: 1.8% [2026-10-20 05:26:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.5172 [2026-10-20 05:26:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0890 [2026-10-20 05:26:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7905 [2026-10-20 05:26:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8138 [2026-10-20 05:26:57][INFO][training.pretrain_engine] [Ph2|Step 270] loss=6.7560 | ppl=859.19 | aux=0.02514 | lr=1.87e-04 | gnorm=1.125 | tok/s=29,316 | MFU=7.0% | step=143073ms | tokens=0.290B | mem=4.2/32.7GB | cost=$141.72/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 00:49 (139.4h) | Phase3: SKIPPED | SFT: Oct 26 05:33 (144.1h) | DPO (Final): Oct 26 06:08 (144.7h) [2026-10-20 05:27:01][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step270 | Tokens: 0.290B | Spent: $141.73 | Remaining: $258.27 [2026-10-20 05:27:01][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step270 [2026-10-20 05:27:01][INFO][training.pretrain_engine] Starting global step 271 (accumulating 32 micro-batches)... [2026-10-20 05:27:01][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 05:27:01][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 05:27:01][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 05:27:01][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 05:27:01][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step270 to Phase-Technologies/forge-500m-pretrain (path: phase2_step270)... [2026-10-20 05:27:01][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 05:27:10][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 05:27:10][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step270 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 05:27:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.5160 [2026-10-20 05:27:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0236 [2026-10-20 05:27:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.5706 [2026-10-20 05:27:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1782 [2026-10-20 05:27:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2933 [2026-10-20 05:27:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4653 [2026-10-20 05:27:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.3662 [2026-10-20 05:27:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8366 [2026-10-20 05:27:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.6701 [2026-10-20 05:27:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.6168 [2026-10-20 05:27:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8807 [2026-10-20 05:27:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.5974 [2026-10-20 05:28:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6619 [2026-10-20 05:28:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3974 [2026-10-20 05:28:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3485 [2026-10-20 05:28:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7787 [2026-10-20 05:28:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0601 [2026-10-20 05:28:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8392 [2026-10-20 05:28:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7003 [2026-10-20 05:28:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6461 [2026-10-20 05:28:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2147 [2026-10-20 05:28:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2410 [2026-10-20 05:28:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9171 [2026-10-20 05:28:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1663 [2026-10-20 05:29:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1856 [2026-10-20 05:29:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3908 [2026-10-20 05:29:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0159 [2026-10-20 05:29:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0102 [2026-10-20 05:29:19][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,570 | MFU: 1.8% [2026-10-20 05:29:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9054 [2026-10-20 05:29:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2618 [2026-10-20 05:29:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9116 [2026-10-20 05:29:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8732 [2026-10-20 05:29:24][INFO][training.pretrain_engine] [Ph2|Step 271] loss=6.8566 | ppl=950.09 | aux=0.02514 | lr=1.87e-04 | gnorm=1.047 | tok/s=29,316 | MFU=7.0% | step=143070ms | tokens=0.292B | mem=4.2/32.7GB | cost=$142.25/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 00:51 (139.4h) | Phase3: SKIPPED | SFT: Oct 26 05:35 (144.1h) | DPO (Final): Oct 26 06:09 (144.7h) [2026-10-20 05:29:24][INFO][training.pretrain_engine] Starting global step 272 (accumulating 32 micro-batches)... [2026-10-20 05:29:24][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 05:29:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4314 [2026-10-20 05:29:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7659 [2026-10-20 05:29:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0518 [2026-10-20 05:29:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8473 [2026-10-20 05:29:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0408 [2026-10-20 05:29:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4414 [2026-10-20 05:29:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.3460 [2026-10-20 05:29:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1421 [2026-10-20 05:30:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4284 [2026-10-20 05:30:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9780 [2026-10-20 05:30:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9453 [2026-10-20 05:30:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3196 [2026-10-20 05:30:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0996 [2026-10-20 05:30:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6459 [2026-10-20 05:30:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5533 [2026-10-20 05:30:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6737 [2026-10-20 05:30:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7087 [2026-10-20 05:30:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3893 [2026-10-20 05:30:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0692 [2026-10-20 05:30:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5138 [2026-10-20 05:31:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5613 [2026-10-20 05:31:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7987 [2026-10-20 05:31:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1622 [2026-10-20 05:31:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.4894 [2026-10-20 05:31:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5293 [2026-10-20 05:31:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5599 [2026-10-20 05:31:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7285 [2026-10-20 05:31:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8292 [2026-10-20 05:31:42][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,600 | MFU: 1.8% [2026-10-20 05:31:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6981 [2026-10-20 05:31:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6587 [2026-10-20 05:31:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4188 [2026-10-20 05:31:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8686 [2026-10-20 05:31:46][INFO][training.pretrain_engine] [Ph2|Step 272] loss=6.8058 | ppl=903.09 | aux=0.02511 | lr=1.88e-04 | gnorm=1.102 | tok/s=29,318 | MFU=7.0% | step=143064ms | tokens=0.293B | mem=4.2/32.7GB | cost=$142.77/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 00:52 (139.3h) | Phase3: SKIPPED | SFT: Oct 26 05:36 (144.1h) | DPO (Final): Oct 26 06:11 (144.7h) [2026-10-20 05:31:46][INFO][training.pretrain_engine] Starting global step 273 (accumulating 32 micro-batches)... [2026-10-20 05:31:46][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 05:32:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.4463 [2026-10-20 05:32:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0234 [2026-10-20 05:32:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.3077 [2026-10-20 05:32:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1477 [2026-10-20 05:32:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7686 [2026-10-20 05:32:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8645 [2026-10-20 05:32:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7494 [2026-10-20 05:32:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5168 [2026-10-20 05:32:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1961 [2026-10-20 05:32:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5916 [2026-10-20 05:32:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6367 [2026-10-20 05:32:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7587 [2026-10-20 05:32:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8935 [2026-10-20 05:32:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8349 [2026-10-20 05:32:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9408 [2026-10-20 05:32:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8612 [2026-10-20 05:33:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2074 [2026-10-20 05:33:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7836 [2026-10-20 05:33:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6407 [2026-10-20 05:33:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3401 [2026-10-20 05:33:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5237 [2026-10-20 05:33:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3971 [2026-10-20 05:33:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0657 [2026-10-20 05:33:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7479 [2026-10-20 05:33:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3742 [2026-10-20 05:33:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8708 [2026-10-20 05:33:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7463 [2026-10-20 05:33:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1445 [2026-10-20 05:34:04][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,612 | MFU: 1.8% [2026-10-20 05:34:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7049 [2026-10-20 05:34:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6713 [2026-10-20 05:34:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2789 [2026-10-20 05:34:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5208 [2026-10-20 05:34:09][INFO][training.pretrain_engine] [Ph2|Step 273] loss=6.8376 | ppl=932.29 | aux=0.02511 | lr=1.89e-04 | gnorm=1.219 | tok/s=29,318 | MFU=7.0% | step=143063ms | tokens=0.294B | mem=4.2/32.7GB | cost=$143.30/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 00:54 (139.3h) | Phase3: SKIPPED | SFT: Oct 26 05:38 (144.1h) | DPO (Final): Oct 26 06:12 (144.6h) [2026-10-20 05:34:09][INFO][training.pretrain_engine] Starting global step 274 (accumulating 32 micro-batches)... [2026-10-20 05:34:09][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 05:34:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9302 [2026-10-20 05:34:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9295 [2026-10-20 05:34:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7798 [2026-10-20 05:34:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7773 [2026-10-20 05:34:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8475 [2026-10-20 05:34:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5613 [2026-10-20 05:34:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8045 [2026-10-20 05:34:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2666 [2026-10-20 05:35:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9179 [2026-10-20 05:35:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1012 [2026-10-20 05:35:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1652 [2026-10-20 05:35:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0128 [2026-10-20 05:35:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2882 [2026-10-20 05:35:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4150 [2026-10-20 05:35:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2971 [2026-10-20 05:35:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1819 [2026-10-20 05:35:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0020 [2026-10-20 05:35:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6711 [2026-10-20 05:35:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4125 [2026-10-20 05:35:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4055 [2026-10-20 05:35:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9034 [2026-10-20 05:35:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7536 [2026-10-20 05:35:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.8793 [2026-10-20 05:35:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5483 [2026-10-20 05:36:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7376 [2026-10-20 05:36:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7880 [2026-10-20 05:36:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8634 [2026-10-20 05:36:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5293 [2026-10-20 05:36:28][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,565 | MFU: 1.8% [2026-10-20 05:36:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.6905 [2026-10-20 05:36:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9781 [2026-10-20 05:36:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9499 [2026-10-20 05:36:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.1208 [2026-10-20 05:36:32][INFO][training.pretrain_engine] [Ph2|Step 274] loss=6.6998 | ppl=812.26 | aux=0.02515 | lr=1.89e-04 | gnorm=0.766 | tok/s=29,318 | MFU=7.0% | step=143064ms | tokens=0.295B | mem=4.2/32.7GB | cost=$143.82/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 00:56 (139.3h) | Phase3: SKIPPED | SFT: Oct 26 05:40 (144.1h) | DPO (Final): Oct 26 06:14 (144.6h) [2026-10-20 05:36:32][INFO][training.pretrain_engine] Starting global step 275 (accumulating 32 micro-batches)... [2026-10-20 05:36:32][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 05:36:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6696 [2026-10-20 05:36:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.5200 [2026-10-20 05:36:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9553 [2026-10-20 05:36:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5716 [2026-10-20 05:37:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5850 [2026-10-20 05:37:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5485 [2026-10-20 05:37:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8854 [2026-10-20 05:37:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7741 [2026-10-20 05:37:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8876 [2026-10-20 05:37:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8795 [2026-10-20 05:37:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9932 [2026-10-20 05:37:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8508 [2026-10-20 05:37:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6874 [2026-10-20 05:37:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9490 [2026-10-20 05:37:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9883 [2026-10-20 05:37:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9105 [2026-10-20 05:38:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1729 [2026-10-20 05:38:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2447 [2026-10-20 05:38:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9512 [2026-10-20 05:38:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0658 [2026-10-20 05:38:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7373 [2026-10-20 05:38:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8927 [2026-10-20 05:38:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6059 [2026-10-20 05:38:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8769 [2026-10-20 05:38:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3622 [2026-10-20 05:38:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9608 [2026-10-20 05:38:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1077 [2026-10-20 05:38:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5892 [2026-10-20 05:38:51][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,569 | MFU: 1.8% [2026-10-20 05:38:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3228 [2026-10-20 05:38:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5934 [2026-10-20 05:38:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3189 [2026-10-20 05:38:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5459 [2026-10-20 05:38:56][INFO][training.pretrain_engine] [Ph2|Step 275] loss=6.7952 | ppl=893.53 | aux=0.02514 | lr=1.90e-04 | gnorm=1.469 | tok/s=29,317 | MFU=7.0% | step=143067ms | tokens=0.296B | mem=4.2/32.7GB | cost=$144.34/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 00:58 (139.3h) | Phase3: SKIPPED | SFT: Oct 26 05:42 (144.1h) | DPO (Final): Oct 26 06:16 (144.6h) [2026-10-20 05:38:56][INFO][training.pretrain_engine] Starting global step 276 (accumulating 32 micro-batches)... [2026-10-20 05:38:56][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 05:39:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1023 [2026-10-20 05:39:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5403 [2026-10-20 05:39:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9308 [2026-10-20 05:39:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6392 [2026-10-20 05:39:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8993 [2026-10-20 05:39:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6180 [2026-10-20 05:39:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9295 [2026-10-20 05:39:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.3794 [2026-10-20 05:39:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9654 [2026-10-20 05:39:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1165 [2026-10-20 05:39:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8206 [2026-10-20 05:39:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6688 [2026-10-20 05:40:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6510 [2026-10-20 05:40:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6780 [2026-10-20 05:40:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5626 [2026-10-20 05:40:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6774 [2026-10-20 05:40:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4277 [2026-10-20 05:40:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5867 [2026-10-20 05:40:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6427 [2026-10-20 05:40:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3400 [2026-10-20 05:40:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2858 [2026-10-20 05:40:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5926 [2026-10-20 05:40:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7094 [2026-10-20 05:40:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.3887 [2026-10-20 05:41:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9243 [2026-10-20 05:41:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9753 [2026-10-20 05:41:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6447 [2026-10-20 05:41:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6240 [2026-10-20 05:41:14][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,576 | MFU: 1.8% [2026-10-20 05:41:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3592 [2026-10-20 05:41:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6068 [2026-10-20 05:41:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7616 [2026-10-20 05:41:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5959 [2026-10-20 05:41:19][INFO][training.pretrain_engine] [Ph2|Step 276] loss=6.8153 | ppl=911.72 | aux=0.02512 | lr=1.90e-04 | gnorm=0.715 | tok/s=29,318 | MFU=7.0% | step=143064ms | tokens=0.297B | mem=4.2/32.7GB | cost=$144.87/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 00:59 (139.3h) | Phase3: SKIPPED | SFT: Oct 26 05:44 (144.0h) | DPO (Final): Oct 26 06:18 (144.6h) [2026-10-20 05:41:19][INFO][training.pretrain_engine] Starting global step 277 (accumulating 32 micro-batches)... [2026-10-20 05:41:19][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 05:41:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7206 [2026-10-20 05:41:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2464 [2026-10-20 05:41:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1954 [2026-10-20 05:41:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6518 [2026-10-20 05:41:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1497 [2026-10-20 05:41:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9668 [2026-10-20 05:41:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0068 [2026-10-20 05:41:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0657 [2026-10-20 05:42:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7086 [2026-10-20 05:42:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8389 [2026-10-20 05:42:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7601 [2026-10-20 05:42:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7472 [2026-10-20 05:42:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3403 [2026-10-20 05:42:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6866 [2026-10-20 05:42:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4885 [2026-10-20 05:42:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5683 [2026-10-20 05:42:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1065 [2026-10-20 05:42:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3193 [2026-10-20 05:42:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0485 [2026-10-20 05:42:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2098 [2026-10-20 05:43:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7692 [2026-10-20 05:43:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7579 [2026-10-20 05:43:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8136 [2026-10-20 05:43:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1601 [2026-10-20 05:43:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8456 [2026-10-20 05:43:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3390 [2026-10-20 05:43:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9411 [2026-10-20 05:43:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2884 [2026-10-20 05:43:38][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,548 | MFU: 1.8% [2026-10-20 05:43:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6634 [2026-10-20 05:43:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8524 [2026-10-20 05:43:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7645 [2026-10-20 05:43:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0668 [2026-10-20 05:43:42][INFO][training.pretrain_engine] [Ph2|Step 277] loss=6.9138 | ppl=1006.05 | aux=0.02512 | lr=1.91e-04 | gnorm=1.398 | tok/s=29,317 | MFU=7.0% | step=143066ms | tokens=0.298B | mem=4.2/32.7GB | cost=$145.39/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 01:01 (139.3h) | Phase3: SKIPPED | SFT: Oct 26 05:46 (144.0h) | DPO (Final): Oct 26 06:20 (144.6h) [2026-10-20 05:43:42][INFO][training.pretrain_engine] Starting global step 278 (accumulating 32 micro-batches)... [2026-10-20 05:43:42][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 05:44:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6895 [2026-10-20 05:44:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0718 [2026-10-20 05:44:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.3699 [2026-10-20 05:44:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5990 [2026-10-20 05:44:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7456 [2026-10-20 05:44:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7837 [2026-10-20 05:44:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0007 [2026-10-20 05:44:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2513 [2026-10-20 05:44:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9701 [2026-10-20 05:44:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0686 [2026-10-20 05:44:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8423 [2026-10-20 05:44:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4174 [2026-10-20 05:44:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3502 [2026-10-20 05:44:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4751 [2026-10-20 05:44:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9345 [2026-10-20 05:44:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5496 [2026-10-20 05:45:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8382 [2026-10-20 05:45:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7284 [2026-10-20 05:45:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3888 [2026-10-20 05:45:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8879 [2026-10-20 05:45:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9559 [2026-10-20 05:45:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5740 [2026-10-20 05:45:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7952 [2026-10-20 05:45:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5711 [2026-10-20 05:45:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1033 [2026-10-20 05:45:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1903 [2026-10-20 05:45:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2719 [2026-10-20 05:45:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0987 [2026-10-20 05:46:01][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,536 | MFU: 1.8% [2026-10-20 05:46:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6267 [2026-10-20 05:46:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.8725 [2026-10-20 05:46:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.1606 [2026-10-20 05:46:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.6386 [2026-10-20 05:46:06][INFO][training.pretrain_engine] [Ph2|Step 278] loss=6.8360 | ppl=930.80 | aux=0.02515 | lr=1.91e-04 | gnorm=0.941 | tok/s=29,316 | MFU=7.0% | step=143071ms | tokens=0.299B | mem=4.2/32.7GB | cost=$145.92/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 01:03 (139.3h) | Phase3: SKIPPED | SFT: Oct 26 05:48 (144.0h) | DPO (Final): Oct 26 06:22 (144.6h) [2026-10-20 05:46:06][INFO][training.pretrain_engine] Starting global step 279 (accumulating 32 micro-batches)... [2026-10-20 05:46:06][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 05:46:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1506 [2026-10-20 05:46:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0185 [2026-10-20 05:46:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8999 [2026-10-20 05:46:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0333 [2026-10-20 05:46:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6954 [2026-10-20 05:46:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6739 [2026-10-20 05:46:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7546 [2026-10-20 05:46:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0452 [2026-10-20 05:46:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5335 [2026-10-20 05:46:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5819 [2026-10-20 05:46:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3920 [2026-10-20 05:46:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4021 [2026-10-20 05:47:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5452 [2026-10-20 05:47:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7995 [2026-10-20 05:47:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5131 [2026-10-20 05:47:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3623 [2026-10-20 05:47:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8661 [2026-10-20 05:47:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0855 [2026-10-20 05:47:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7751 [2026-10-20 05:47:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0691 [2026-10-20 05:47:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8419 [2026-10-20 05:47:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6493 [2026-10-20 05:47:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4128 [2026-10-20 05:47:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1930 [2026-10-20 05:48:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1752 [2026-10-20 05:48:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0991 [2026-10-20 05:48:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0158 [2026-10-20 05:48:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2315 [2026-10-20 05:48:24][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,566 | MFU: 1.8% [2026-10-20 05:48:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6454 [2026-10-20 05:48:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5240 [2026-10-20 05:48:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7112 [2026-10-20 05:48:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1219 [2026-10-20 05:48:29][INFO][training.pretrain_engine] [Ph2|Step 279] loss=6.8268 | ppl=922.26 | aux=0.02517 | lr=1.92e-04 | gnorm=2.344 | tok/s=29,316 | MFU=7.0% | step=143071ms | tokens=0.300B | mem=4.2/32.7GB | cost=$146.44/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 01:05 (139.3h) | Phase3: SKIPPED | SFT: Oct 26 05:49 (144.0h) | DPO (Final): Oct 26 06:24 (144.6h) [2026-10-20 05:48:32][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step279 | Tokens: 0.300B | Spent: $146.45 | Remaining: $253.55 [2026-10-20 05:48:32][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step279 [2026-10-20 05:48:32][INFO][training.pretrain_engine] Starting global step 280 (accumulating 32 micro-batches)... [2026-10-20 05:48:32][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 05:48:33][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 05:48:33][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 05:48:33][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 05:48:33][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step279 to Phase-Technologies/forge-500m-pretrain (path: phase2_step279)... [2026-10-20 05:48:33][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 05:48:42][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 05:48:42][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step279 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 05:48:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0223 [2026-10-20 05:48:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8516 [2026-10-20 05:48:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8622 [2026-10-20 05:48:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6475 [2026-10-20 05:49:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6370 [2026-10-20 05:49:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5755 [2026-10-20 05:49:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9779 [2026-10-20 05:49:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5766 [2026-10-20 05:49:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3662 [2026-10-20 05:49:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2066 [2026-10-20 05:49:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4127 [2026-10-20 05:49:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4268 [2026-10-20 05:49:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0118 [2026-10-20 05:49:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5517 [2026-10-20 05:49:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6978 [2026-10-20 05:49:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9213 [2026-10-20 05:49:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0502 [2026-10-20 05:49:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6965 [2026-10-20 05:49:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3857 [2026-10-20 05:50:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3915 [2026-10-20 05:50:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6954 [2026-10-20 05:50:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4123 [2026-10-20 05:50:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5893 [2026-10-20 05:50:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5040 [2026-10-20 05:50:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6546 [2026-10-20 05:50:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1152 [2026-10-20 05:50:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7298 [2026-10-20 05:50:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8255 [2026-10-20 05:50:51][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,567 | MFU: 1.8% [2026-10-20 05:50:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5796 [2026-10-20 05:50:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9748 [2026-10-20 05:50:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7429 [2026-10-20 05:50:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6986 [2026-10-20 05:50:56][INFO][training.pretrain_engine] [Ph2|Step 280] loss=6.8487 | ppl=942.67 | aux=0.02515 | lr=1.93e-04 | gnorm=1.875 | tok/s=29,316 | MFU=7.0% | step=143073ms | tokens=0.301B | mem=4.2/32.7GB | cost=$146.97/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 01:07 (139.3h) | Phase3: SKIPPED | SFT: Oct 26 05:51 (144.0h) | DPO (Final): Oct 26 06:26 (144.6h) [2026-10-20 05:50:56][INFO][training.pretrain_engine] Starting global step 281 (accumulating 32 micro-batches)... [2026-10-20 05:50:56][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 05:51:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7188 [2026-10-20 05:51:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0468 [2026-10-20 05:51:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6552 [2026-10-20 05:51:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2950 [2026-10-20 05:51:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5220 [2026-10-20 05:51:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5310 [2026-10-20 05:51:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7754 [2026-10-20 05:51:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8533 [2026-10-20 05:51:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6965 [2026-10-20 05:51:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3491 [2026-10-20 05:51:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4154 [2026-10-20 05:51:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9833 [2026-10-20 05:52:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8998 [2026-10-20 05:52:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4916 [2026-10-20 05:52:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5539 [2026-10-20 05:52:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4202 [2026-10-20 05:52:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5444 [2026-10-20 05:52:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6507 [2026-10-20 05:52:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3243 [2026-10-20 05:52:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3895 [2026-10-20 05:52:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.3727 [2026-10-20 05:52:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.6035 [2026-10-20 05:52:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9474 [2026-10-20 05:52:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.7991 [2026-10-20 05:53:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9288 [2026-10-20 05:53:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4612 [2026-10-20 05:53:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8134 [2026-10-20 05:53:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8865 [2026-10-20 05:53:14][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,556 | MFU: 1.8% [2026-10-20 05:53:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7684 [2026-10-20 05:53:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7840 [2026-10-20 05:53:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9509 [2026-10-20 05:53:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8826 [2026-10-20 05:53:19][INFO][training.pretrain_engine] [Ph2|Step 281] loss=6.6850 | ppl=800.34 | aux=0.02515 | lr=1.93e-04 | gnorm=1.961 | tok/s=29,315 | MFU=7.0% | step=143078ms | tokens=0.302B | mem=4.2/32.7GB | cost=$147.50/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 01:09 (139.3h) | Phase3: SKIPPED | SFT: Oct 26 05:54 (144.0h) | DPO (Final): Oct 26 06:28 (144.6h) [2026-10-20 05:53:19][INFO][training.pretrain_engine] Starting global step 282 (accumulating 32 micro-batches)... [2026-10-20 05:53:19][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 05:53:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7412 [2026-10-20 05:53:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7035 [2026-10-20 05:53:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5468 [2026-10-20 05:53:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4251 [2026-10-20 05:53:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6380 [2026-10-20 05:53:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4098 [2026-10-20 05:53:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2815 [2026-10-20 05:53:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6696 [2026-10-20 05:54:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5061 [2026-10-20 05:54:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5809 [2026-10-20 05:54:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5459 [2026-10-20 05:54:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6446 [2026-10-20 05:54:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2619 [2026-10-20 05:54:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3344 [2026-10-20 05:54:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1574 [2026-10-20 05:54:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4663 [2026-10-20 05:54:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6942 [2026-10-20 05:54:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6373 [2026-10-20 05:54:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6638 [2026-10-20 05:54:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3701 [2026-10-20 05:55:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2248 [2026-10-20 05:55:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0103 [2026-10-20 05:55:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5030 [2026-10-20 05:55:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8886 [2026-10-20 05:55:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8060 [2026-10-20 05:55:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6200 [2026-10-20 05:55:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4814 [2026-10-20 05:55:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6453 [2026-10-20 05:55:37][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,566 | MFU: 1.8% [2026-10-20 05:55:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9959 [2026-10-20 05:55:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6568 [2026-10-20 05:55:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6711 [2026-10-20 05:55:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5729 [2026-10-20 05:55:42][INFO][training.pretrain_engine] [Ph2|Step 282] loss=6.6308 | ppl=758.07 | aux=0.02515 | lr=1.94e-04 | gnorm=1.664 | tok/s=29,315 | MFU=7.0% | step=143075ms | tokens=0.303B | mem=4.2/32.7GB | cost=$148.02/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 01:11 (139.3h) | Phase3: SKIPPED | SFT: Oct 26 05:55 (144.0h) | DPO (Final): Oct 26 06:29 (144.6h) [2026-10-20 05:55:42][INFO][training.pretrain_engine] Starting global step 283 (accumulating 32 micro-batches)... [2026-10-20 05:55:42][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 05:56:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9046 [2026-10-20 05:56:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2096 [2026-10-20 05:56:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8266 [2026-10-20 05:56:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3974 [2026-10-20 05:56:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8690 [2026-10-20 05:56:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7401 [2026-10-20 05:56:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8803 [2026-10-20 05:56:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4977 [2026-10-20 05:56:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9785 [2026-10-20 05:56:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4936 [2026-10-20 05:56:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5054 [2026-10-20 05:56:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9766 [2026-10-20 05:56:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5934 [2026-10-20 05:56:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4206 [2026-10-20 05:56:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6384 [2026-10-20 05:56:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6965 [2026-10-20 05:57:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0871 [2026-10-20 05:57:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7756 [2026-10-20 05:57:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8110 [2026-10-20 05:57:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6844 [2026-10-20 05:57:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1797 [2026-10-20 05:57:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8816 [2026-10-20 05:57:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8012 [2026-10-20 05:57:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0971 [2026-10-20 05:57:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0868 [2026-10-20 05:57:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1512 [2026-10-20 05:57:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0167 [2026-10-20 05:57:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2658 [2026-10-20 05:58:01][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,562 | MFU: 1.8% [2026-10-20 05:58:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.1948 [2026-10-20 05:58:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2195 [2026-10-20 05:58:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8126 [2026-10-20 05:58:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8879 [2026-10-20 05:58:05][INFO][training.pretrain_engine] [Ph2|Step 283] loss=6.6610 | ppl=781.33 | aux=0.02514 | lr=1.94e-04 | gnorm=1.227 | tok/s=29,315 | MFU=7.0% | step=143077ms | tokens=0.304B | mem=4.2/32.7GB | cost=$148.55/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 01:13 (139.3h) | Phase3: SKIPPED | SFT: Oct 26 05:57 (144.0h) | DPO (Final): Oct 26 06:31 (144.6h) [2026-10-20 05:58:05][INFO][training.pretrain_engine] Starting global step 284 (accumulating 32 micro-batches)... [2026-10-20 05:58:05][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 05:58:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9694 [2026-10-20 05:58:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5426 [2026-10-20 05:58:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5590 [2026-10-20 05:58:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0240 [2026-10-20 05:58:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1147 [2026-10-20 05:58:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0066 [2026-10-20 05:58:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7882 [2026-10-20 05:58:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9606 [2026-10-20 05:58:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5929 [2026-10-20 05:58:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5141 [2026-10-20 05:58:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8669 [2026-10-20 05:58:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0179 [2026-10-20 05:59:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4692 [2026-10-20 05:59:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2128 [2026-10-20 05:59:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9292 [2026-10-20 05:59:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7807 [2026-10-20 05:59:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9991 [2026-10-20 05:59:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9495 [2026-10-20 05:59:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7152 [2026-10-20 05:59:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7754 [2026-10-20 05:59:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2976 [2026-10-20 05:59:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4605 [2026-10-20 05:59:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7665 [2026-10-20 05:59:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2084 [2026-10-20 06:00:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2512 [2026-10-20 06:00:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6979 [2026-10-20 06:00:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9273 [2026-10-20 06:00:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6584 [2026-10-20 06:00:24][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,557 | MFU: 1.8% [2026-10-20 06:00:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0879 [2026-10-20 06:00:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2562 [2026-10-20 06:00:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9322 [2026-10-20 06:00:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0001 [2026-10-20 06:00:29][INFO][training.pretrain_engine] [Ph2|Step 284] loss=6.7440 | ppl=848.95 | aux=0.02517 | lr=1.95e-04 | gnorm=2.188 | tok/s=29,314 | MFU=7.0% | step=143080ms | tokens=0.305B | mem=4.2/32.7GB | cost=$149.07/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 01:15 (139.2h) | Phase3: SKIPPED | SFT: Oct 26 05:59 (144.0h) | DPO (Final): Oct 26 06:33 (144.6h) [2026-10-20 06:00:29][INFO][training.pretrain_engine] Starting global step 285 (accumulating 32 micro-batches)... [2026-10-20 06:00:29][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 06:00:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2594 [2026-10-20 06:00:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7847 [2026-10-20 06:00:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7177 [2026-10-20 06:00:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5615 [2026-10-20 06:01:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6350 [2026-10-20 06:01:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4182 [2026-10-20 06:01:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6969 [2026-10-20 06:01:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1045 [2026-10-20 06:01:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6528 [2026-10-20 06:01:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4735 [2026-10-20 06:01:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0502 [2026-10-20 06:01:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7647 [2026-10-20 06:01:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9963 [2026-10-20 06:01:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1370 [2026-10-20 06:01:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0028 [2026-10-20 06:01:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0125 [2026-10-20 06:01:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0028 [2026-10-20 06:01:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9904 [2026-10-20 06:01:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4549 [2026-10-20 06:01:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9408 [2026-10-20 06:02:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6569 [2026-10-20 06:02:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6315 [2026-10-20 06:02:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4512 [2026-10-20 06:02:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3720 [2026-10-20 06:02:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1163 [2026-10-20 06:02:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8299 [2026-10-20 06:02:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7960 [2026-10-20 06:02:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7376 [2026-10-20 06:02:47][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,588 | MFU: 1.8% [2026-10-20 06:02:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5872 [2026-10-20 06:02:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6378 [2026-10-20 06:02:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7125 [2026-10-20 06:02:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8026 [2026-10-20 06:02:52][INFO][training.pretrain_engine] [Ph2|Step 285] loss=6.7999 | ppl=897.72 | aux=0.02517 | lr=1.95e-04 | gnorm=1.820 | tok/s=29,314 | MFU=7.0% | step=143080ms | tokens=0.306B | mem=4.2/32.7GB | cost=$149.59/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 01:17 (139.2h) | Phase3: SKIPPED | SFT: Oct 26 06:01 (144.0h) | DPO (Final): Oct 26 06:35 (144.5h) [2026-10-20 06:02:52][INFO][training.pretrain_engine] Starting global step 286 (accumulating 32 micro-batches)... [2026-10-20 06:02:52][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 06:03:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8192 [2026-10-20 06:03:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5475 [2026-10-20 06:03:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2861 [2026-10-20 06:03:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8861 [2026-10-20 06:03:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0489 [2026-10-20 06:03:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8581 [2026-10-20 06:03:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8225 [2026-10-20 06:03:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.6676 [2026-10-20 06:03:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7993 [2026-10-20 06:03:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0343 [2026-10-20 06:03:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0697 [2026-10-20 06:03:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8073 [2026-10-20 06:04:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5242 [2026-10-20 06:04:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5560 [2026-10-20 06:04:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2850 [2026-10-20 06:04:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7915 [2026-10-20 06:04:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8813 [2026-10-20 06:04:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8932 [2026-10-20 06:04:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0229 [2026-10-20 06:04:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0809 [2026-10-20 06:04:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2352 [2026-10-20 06:04:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0776 [2026-10-20 06:04:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0841 [2026-10-20 06:04:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2102 [2026-10-20 06:04:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7803 [2026-10-20 06:04:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8403 [2026-10-20 06:04:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0582 [2026-10-20 06:04:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2301 [2026-10-20 06:05:11][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,540 | MFU: 1.8% [2026-10-20 06:05:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2153 [2026-10-20 06:05:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4553 [2026-10-20 06:05:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4117 [2026-10-20 06:05:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4105 [2026-10-20 06:05:15][INFO][training.pretrain_engine] [Ph2|Step 286] loss=6.8250 | ppl=920.57 | aux=0.02519 | lr=1.96e-04 | gnorm=2.062 | tok/s=29,314 | MFU=7.0% | step=143083ms | tokens=0.307B | mem=4.2/32.7GB | cost=$150.12/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 01:18 (139.2h) | Phase3: SKIPPED | SFT: Oct 26 06:03 (144.0h) | DPO (Final): Oct 26 06:37 (144.5h) [2026-10-20 06:05:15][INFO][training.pretrain_engine] Starting global step 287 (accumulating 32 micro-batches)... [2026-10-20 06:05:15][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 06:05:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5714 [2026-10-20 06:05:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6716 [2026-10-20 06:05:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8447 [2026-10-20 06:05:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7378 [2026-10-20 06:05:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8042 [2026-10-20 06:05:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3111 [2026-10-20 06:05:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7621 [2026-10-20 06:05:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3727 [2026-10-20 06:06:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5432 [2026-10-20 06:06:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6165 [2026-10-20 06:06:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1112 [2026-10-20 06:06:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2651 [2026-10-20 06:06:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5736 [2026-10-20 06:06:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9268 [2026-10-20 06:06:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7361 [2026-10-20 06:06:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0697 [2026-10-20 06:06:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6160 [2026-10-20 06:06:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9689 [2026-10-20 06:06:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1113 [2026-10-20 06:06:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7676 [2026-10-20 06:07:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5679 [2026-10-20 06:07:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5513 [2026-10-20 06:07:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5888 [2026-10-20 06:07:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5034 [2026-10-20 06:07:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6111 [2026-10-20 06:07:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3457 [2026-10-20 06:07:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5691 [2026-10-20 06:07:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8884 [2026-10-20 06:07:34][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,559 | MFU: 1.8% [2026-10-20 06:07:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5260 [2026-10-20 06:07:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5117 [2026-10-20 06:07:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6077 [2026-10-20 06:07:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4514 [2026-10-20 06:07:38][INFO][training.pretrain_engine] [Ph2|Step 287] loss=6.6966 | ppl=809.67 | aux=0.02515 | lr=1.96e-04 | gnorm=1.203 | tok/s=29,313 | MFU=7.0% | step=143085ms | tokens=0.308B | mem=4.2/32.7GB | cost=$150.64/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 01:20 (139.2h) | Phase3: SKIPPED | SFT: Oct 26 06:05 (144.0h) | DPO (Final): Oct 26 06:39 (144.5h) [2026-10-20 06:07:38][INFO][training.pretrain_engine] Starting global step 288 (accumulating 32 micro-batches)... [2026-10-20 06:07:38][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 06:07:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8893 [2026-10-20 06:07:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9514 [2026-10-20 06:07:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8541 [2026-10-20 06:07:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6635 [2026-10-20 06:08:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8801 [2026-10-20 06:08:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0268 [2026-10-20 06:08:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0522 [2026-10-20 06:08:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8195 [2026-10-20 06:08:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8024 [2026-10-20 06:08:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5199 [2026-10-20 06:08:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8805 [2026-10-20 06:08:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9152 [2026-10-20 06:08:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5640 [2026-10-20 06:08:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5292 [2026-10-20 06:08:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7949 [2026-10-20 06:08:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5191 [2026-10-20 06:09:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1889 [2026-10-20 06:09:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6151 [2026-10-20 06:09:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4808 [2026-10-20 06:09:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6838 [2026-10-20 06:09:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7115 [2026-10-20 06:09:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0581 [2026-10-20 06:09:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9449 [2026-10-20 06:09:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6499 [2026-10-20 06:09:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9673 [2026-10-20 06:09:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2619 [2026-10-20 06:09:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5984 [2026-10-20 06:09:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.4019 [2026-10-20 06:09:57][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,561 | MFU: 1.8% [2026-10-20 06:10:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2981 [2026-10-20 06:10:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3908 [2026-10-20 06:10:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4992 [2026-10-20 06:10:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4869 [2026-10-20 06:10:02][INFO][training.pretrain_engine] [Ph2|Step 288] loss=6.6491 | ppl=772.08 | aux=0.02514 | lr=1.97e-04 | gnorm=1.930 | tok/s=29,313 | MFU=7.0% | step=143085ms | tokens=0.309B | mem=4.2/32.7GB | cost=$151.16/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 01:22 (139.2h) | Phase3: SKIPPED | SFT: Oct 26 06:07 (143.9h) | DPO (Final): Oct 26 06:41 (144.5h) [2026-10-20 06:10:05][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step288 | Tokens: 0.309B | Spent: $151.18 | Remaining: $248.82 [2026-10-20 06:10:05][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step288 [2026-10-20 06:10:05][INFO][training.pretrain_engine] Starting global step 289 (accumulating 32 micro-batches)... [2026-10-20 06:10:05][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 06:10:05][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 06:10:05][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 06:10:05][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 06:10:05][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step288 to Phase-Technologies/forge-500m-pretrain (path: phase2_step288)... [2026-10-20 06:10:05][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 06:10:15][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 06:10:15][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step288 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 06:10:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1378 [2026-10-20 06:10:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5571 [2026-10-20 06:10:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5313 [2026-10-20 06:10:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6065 [2026-10-20 06:10:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7232 [2026-10-20 06:10:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2120 [2026-10-20 06:10:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0901 [2026-10-20 06:10:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3476 [2026-10-20 06:10:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5620 [2026-10-20 06:10:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2377 [2026-10-20 06:10:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9700 [2026-10-20 06:10:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3104 [2026-10-20 06:11:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4392 [2026-10-20 06:11:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8745 [2026-10-20 06:11:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4191 [2026-10-20 06:11:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7759 [2026-10-20 06:11:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8134 [2026-10-20 06:11:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6575 [2026-10-20 06:11:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1751 [2026-10-20 06:11:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7091 [2026-10-20 06:11:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0139 [2026-10-20 06:11:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.5524 [2026-10-20 06:11:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2393 [2026-10-20 06:11:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0789 [2026-10-20 06:12:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5393 [2026-10-20 06:12:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7085 [2026-10-20 06:12:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5088 [2026-10-20 06:12:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7772 [2026-10-20 06:12:24][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,570 | MFU: 1.8% [2026-10-20 06:12:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8902 [2026-10-20 06:12:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.1192 [2026-10-20 06:12:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5515 [2026-10-20 06:12:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6512 [2026-10-20 06:12:28][INFO][training.pretrain_engine] [Ph2|Step 289] loss=6.7909 | ppl=889.69 | aux=0.02511 | lr=1.98e-04 | gnorm=1.094 | tok/s=29,314 | MFU=7.0% | step=143080ms | tokens=0.310B | mem=4.2/32.7GB | cost=$151.70/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 01:24 (139.2h) | Phase3: SKIPPED | SFT: Oct 26 06:08 (143.9h) | DPO (Final): Oct 26 06:42 (144.5h) [2026-10-20 06:12:28][INFO][training.pretrain_engine] Starting global step 290 (accumulating 32 micro-batches)... [2026-10-20 06:12:28][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 06:12:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0541 [2026-10-20 06:12:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5967 [2026-10-20 06:12:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7805 [2026-10-20 06:12:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7545 [2026-10-20 06:13:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3619 [2026-10-20 06:13:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8216 [2026-10-20 06:13:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0940 [2026-10-20 06:13:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2898 [2026-10-20 06:13:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.2493 [2026-10-20 06:13:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1215 [2026-10-20 06:13:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5162 [2026-10-20 06:13:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4112 [2026-10-20 06:13:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8156 [2026-10-20 06:13:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0206 [2026-10-20 06:13:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0989 [2026-10-20 06:13:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8502 [2026-10-20 06:13:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1858 [2026-10-20 06:13:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6836 [2026-10-20 06:13:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7115 [2026-10-20 06:13:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5267 [2026-10-20 06:14:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6476 [2026-10-20 06:14:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6138 [2026-10-20 06:14:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4871 [2026-10-20 06:14:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5929 [2026-10-20 06:14:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4632 [2026-10-20 06:14:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3656 [2026-10-20 06:14:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4712 [2026-10-20 06:14:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5847 [2026-10-20 06:14:46][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,575 | MFU: 1.8% [2026-10-20 06:14:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.7709 [2026-10-20 06:14:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8171 [2026-10-20 06:14:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.5003 [2026-10-20 06:14:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6693 [2026-10-20 06:14:51][INFO][training.pretrain_engine] [Ph2|Step 290] loss=6.7742 | ppl=874.94 | aux=0.02517 | lr=1.98e-04 | gnorm=2.359 | tok/s=29,316 | MFU=7.0% | step=143073ms | tokens=0.311B | mem=4.2/32.7GB | cost=$152.22/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 01:25 (139.2h) | Phase3: SKIPPED | SFT: Oct 26 06:09 (143.9h) | DPO (Final): Oct 26 06:44 (144.5h) [2026-10-20 06:14:51][INFO][training.pretrain_engine] Starting global step 291 (accumulating 32 micro-batches)... [2026-10-20 06:14:51][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 06:15:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8401 [2026-10-20 06:15:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8722 [2026-10-20 06:15:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1739 [2026-10-20 06:15:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8822 [2026-10-20 06:15:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9475 [2026-10-20 06:15:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7323 [2026-10-20 06:15:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2935 [2026-10-20 06:15:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7190 [2026-10-20 06:15:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6153 [2026-10-20 06:15:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6297 [2026-10-20 06:15:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8549 [2026-10-20 06:15:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7399 [2026-10-20 06:16:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5029 [2026-10-20 06:16:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6637 [2026-10-20 06:16:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6477 [2026-10-20 06:16:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.3866 [2026-10-20 06:16:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4443 [2026-10-20 06:16:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7076 [2026-10-20 06:16:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6790 [2026-10-20 06:16:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5880 [2026-10-20 06:16:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2543 [2026-10-20 06:16:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0859 [2026-10-20 06:16:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8052 [2026-10-20 06:16:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3611 [2026-10-20 06:16:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0275 [2026-10-20 06:16:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0209 [2026-10-20 06:16:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0764 [2026-10-20 06:16:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.4289 [2026-10-20 06:17:10][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,545 | MFU: 1.8% [2026-10-20 06:17:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7094 [2026-10-20 06:17:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5537 [2026-10-20 06:17:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4481 [2026-10-20 06:17:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4954 [2026-10-20 06:17:15][INFO][training.pretrain_engine] [Ph2|Step 291] loss=6.7710 | ppl=872.19 | aux=0.02518 | lr=1.99e-04 | gnorm=1.930 | tok/s=29,315 | MFU=7.0% | step=143075ms | tokens=0.312B | mem=4.2/32.7GB | cost=$152.75/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 01:27 (139.2h) | Phase3: SKIPPED | SFT: Oct 26 06:11 (143.9h) | DPO (Final): Oct 26 06:45 (144.5h) [2026-10-20 06:17:15][INFO][training.pretrain_engine] Starting global step 292 (accumulating 32 micro-batches)... [2026-10-20 06:17:15][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 06:17:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7958 [2026-10-20 06:17:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5000 [2026-10-20 06:17:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7938 [2026-10-20 06:17:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3625 [2026-10-20 06:17:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7878 [2026-10-20 06:17:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3916 [2026-10-20 06:17:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7782 [2026-10-20 06:17:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5146 [2026-10-20 06:18:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1045 [2026-10-20 06:18:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7697 [2026-10-20 06:18:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1050 [2026-10-20 06:18:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7732 [2026-10-20 06:18:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9121 [2026-10-20 06:18:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0153 [2026-10-20 06:18:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9288 [2026-10-20 06:18:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2051 [2026-10-20 06:18:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6424 [2026-10-20 06:18:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7006 [2026-10-20 06:18:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6700 [2026-10-20 06:18:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7978 [2026-10-20 06:19:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.9603 [2026-10-20 06:19:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5873 [2026-10-20 06:19:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1080 [2026-10-20 06:19:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.9514 [2026-10-20 06:19:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6670 [2026-10-20 06:19:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1481 [2026-10-20 06:19:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1243 [2026-10-20 06:19:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2334 [2026-10-20 06:19:34][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,554 | MFU: 1.8% [2026-10-20 06:19:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7281 [2026-10-20 06:19:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.7209 [2026-10-20 06:19:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8543 [2026-10-20 06:19:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9213 [2026-10-20 06:19:38][INFO][training.pretrain_engine] [Ph2|Step 292] loss=6.7892 | ppl=888.19 | aux=0.02519 | lr=1.99e-04 | gnorm=1.898 | tok/s=29,315 | MFU=7.0% | step=143076ms | tokens=0.314B | mem=4.2/32.7GB | cost=$153.27/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 01:29 (139.2h) | Phase3: SKIPPED | SFT: Oct 26 06:13 (143.9h) | DPO (Final): Oct 26 06:47 (144.5h) [2026-10-20 06:19:38][INFO][training.pretrain_engine] Starting global step 293 (accumulating 32 micro-batches)... [2026-10-20 06:19:38][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 06:19:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6057 [2026-10-20 06:19:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8414 [2026-10-20 06:19:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5904 [2026-10-20 06:19:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5957 [2026-10-20 06:20:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4656 [2026-10-20 06:20:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1577 [2026-10-20 06:20:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5594 [2026-10-20 06:20:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9258 [2026-10-20 06:20:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7327 [2026-10-20 06:20:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8529 [2026-10-20 06:20:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9397 [2026-10-20 06:20:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7673 [2026-10-20 06:20:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7415 [2026-10-20 06:20:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8835 [2026-10-20 06:20:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5634 [2026-10-20 06:20:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3955 [2026-10-20 06:21:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4933 [2026-10-20 06:21:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2754 [2026-10-20 06:21:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1453 [2026-10-20 06:21:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6884 [2026-10-20 06:21:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3248 [2026-10-20 06:21:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8127 [2026-10-20 06:21:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7570 [2026-10-20 06:21:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4911 [2026-10-20 06:21:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9951 [2026-10-20 06:21:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3533 [2026-10-20 06:21:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6851 [2026-10-20 06:21:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7022 [2026-10-20 06:21:57][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,579 | MFU: 1.8% [2026-10-20 06:22:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7530 [2026-10-20 06:22:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0650 [2026-10-20 06:22:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9098 [2026-10-20 06:22:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0761 [2026-10-20 06:22:02][INFO][training.pretrain_engine] [Ph2|Step 293] loss=6.7903 | ppl=889.20 | aux=0.02515 | lr=2.00e-04 | gnorm=1.766 | tok/s=29,315 | MFU=7.0% | step=143076ms | tokens=0.315B | mem=4.2/32.7GB | cost=$153.80/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 01:31 (139.2h) | Phase3: SKIPPED | SFT: Oct 26 06:15 (143.9h) | DPO (Final): Oct 26 06:49 (144.5h) [2026-10-20 06:22:02][INFO][training.pretrain_engine] Starting global step 294 (accumulating 32 micro-batches)... [2026-10-20 06:22:02][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 06:22:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8658 [2026-10-20 06:22:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5808 [2026-10-20 06:22:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7894 [2026-10-20 06:22:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8458 [2026-10-20 06:22:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8246 [2026-10-20 06:22:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9085 [2026-10-20 06:22:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3133 [2026-10-20 06:22:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0059 [2026-10-20 06:22:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6180 [2026-10-20 06:22:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1615 [2026-10-20 06:22:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4422 [2026-10-20 06:22:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1971 [2026-10-20 06:23:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8613 [2026-10-20 06:23:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7388 [2026-10-20 06:23:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7350 [2026-10-20 06:23:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5754 [2026-10-20 06:23:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6549 [2026-10-20 06:23:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0079 [2026-10-20 06:23:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0307 [2026-10-20 06:23:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1562 [2026-10-20 06:23:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6119 [2026-10-20 06:23:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3363 [2026-10-20 06:23:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4493 [2026-10-20 06:23:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2885 [2026-10-20 06:24:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5934 [2026-10-20 06:24:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4335 [2026-10-20 06:24:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3620 [2026-10-20 06:24:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4811 [2026-10-20 06:24:20][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,551 | MFU: 1.8% [2026-10-20 06:24:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=8.0243 [2026-10-20 06:24:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8571 [2026-10-20 06:24:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3113 [2026-10-20 06:24:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.1020 [2026-10-20 06:24:25][INFO][training.pretrain_engine] [Ph2|Step 294] loss=6.8041 | ppl=901.54 | aux=0.02516 | lr=2.00e-04 | gnorm=1.070 | tok/s=29,314 | MFU=7.0% | step=143080ms | tokens=0.316B | mem=4.2/32.7GB | cost=$154.32/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 01:33 (139.1h) | Phase3: SKIPPED | SFT: Oct 26 06:17 (143.9h) | DPO (Final): Oct 26 06:51 (144.5h) [2026-10-20 06:24:25][INFO][training.pretrain_engine] Starting global step 295 (accumulating 32 micro-batches)... [2026-10-20 06:24:25][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 06:24:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7473 [2026-10-20 06:24:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6916 [2026-10-20 06:24:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7860 [2026-10-20 06:24:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.4207 [2026-10-20 06:25:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6623 [2026-10-20 06:25:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2211 [2026-10-20 06:25:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5646 [2026-10-20 06:25:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5135 [2026-10-20 06:25:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9114 [2026-10-20 06:25:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6520 [2026-10-20 06:25:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8614 [2026-10-20 06:25:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1471 [2026-10-20 06:25:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8258 [2026-10-20 06:25:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7176 [2026-10-20 06:25:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7362 [2026-10-20 06:25:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1572 [2026-10-20 06:25:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4410 [2026-10-20 06:25:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8256 [2026-10-20 06:25:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7595 [2026-10-20 06:25:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2109 [2026-10-20 06:26:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6165 [2026-10-20 06:26:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8277 [2026-10-20 06:26:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7589 [2026-10-20 06:26:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5190 [2026-10-20 06:26:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0468 [2026-10-20 06:26:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9659 [2026-10-20 06:26:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3043 [2026-10-20 06:26:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8754 [2026-10-20 06:26:43][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,572 | MFU: 1.8% [2026-10-20 06:26:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5102 [2026-10-20 06:26:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7092 [2026-10-20 06:26:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0122 [2026-10-20 06:26:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3475 [2026-10-20 06:26:48][INFO][training.pretrain_engine] [Ph2|Step 295] loss=6.7896 | ppl=888.53 | aux=0.02514 | lr=2.01e-04 | gnorm=1.766 | tok/s=29,313 | MFU=7.0% | step=143088ms | tokens=0.317B | mem=4.2/32.7GB | cost=$154.84/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 01:35 (139.1h) | Phase3: SKIPPED | SFT: Oct 26 06:19 (143.9h) | DPO (Final): Oct 26 06:53 (144.5h) [2026-10-20 06:26:48][INFO][training.pretrain_engine] Starting global step 296 (accumulating 32 micro-batches)... [2026-10-20 06:26:48][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 06:27:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4735 [2026-10-20 06:27:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7268 [2026-10-20 06:27:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6381 [2026-10-20 06:27:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4647 [2026-10-20 06:27:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6781 [2026-10-20 06:27:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3264 [2026-10-20 06:27:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2400 [2026-10-20 06:27:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3352 [2026-10-20 06:27:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9403 [2026-10-20 06:27:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1814 [2026-10-20 06:27:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2491 [2026-10-20 06:27:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9589 [2026-10-20 06:28:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6419 [2026-10-20 06:28:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7165 [2026-10-20 06:28:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6952 [2026-10-20 06:28:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8415 [2026-10-20 06:28:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9574 [2026-10-20 06:28:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7123 [2026-10-20 06:28:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4936 [2026-10-20 06:28:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4265 [2026-10-20 06:28:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9916 [2026-10-20 06:28:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1820 [2026-10-20 06:28:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6197 [2026-10-20 06:28:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6482 [2026-10-20 06:28:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9183 [2026-10-20 06:28:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6110 [2026-10-20 06:28:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7158 [2026-10-20 06:28:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6994 [2026-10-20 06:29:07][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,546 | MFU: 1.8% [2026-10-20 06:29:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4498 [2026-10-20 06:29:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.5064 [2026-10-20 06:29:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9991 [2026-10-20 06:29:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8112 [2026-10-20 06:29:12][INFO][training.pretrain_engine] [Ph2|Step 296] loss=6.5982 | ppl=733.78 | aux=0.02514 | lr=2.02e-04 | gnorm=1.047 | tok/s=29,312 | MFU=7.0% | step=143094ms | tokens=0.318B | mem=4.2/32.7GB | cost=$155.37/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 01:37 (139.1h) | Phase3: SKIPPED | SFT: Oct 26 06:21 (143.9h) | DPO (Final): Oct 26 06:56 (144.4h) [2026-10-20 06:29:12][INFO][training.pretrain_engine] Starting global step 297 (accumulating 32 micro-batches)... [2026-10-20 06:29:12][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 06:29:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5038 [2026-10-20 06:29:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5217 [2026-10-20 06:29:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6560 [2026-10-20 06:29:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6570 [2026-10-20 06:29:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4658 [2026-10-20 06:29:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0124 [2026-10-20 06:29:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6950 [2026-10-20 06:29:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8751 [2026-10-20 06:30:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7351 [2026-10-20 06:30:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1674 [2026-10-20 06:30:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3690 [2026-10-20 06:30:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6450 [2026-10-20 06:30:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6858 [2026-10-20 06:30:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8218 [2026-10-20 06:30:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8232 [2026-10-20 06:30:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7614 [2026-10-20 06:30:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8149 [2026-10-20 06:30:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2094 [2026-10-20 06:30:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6984 [2026-10-20 06:30:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4228 [2026-10-20 06:30:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0273 [2026-10-20 06:30:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0218 [2026-10-20 06:30:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3496 [2026-10-20 06:30:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0170 [2026-10-20 06:31:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2564 [2026-10-20 06:31:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3194 [2026-10-20 06:31:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1671 [2026-10-20 06:31:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3020 [2026-10-20 06:31:30][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,574 | MFU: 1.8% [2026-10-20 06:31:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5579 [2026-10-20 06:31:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3792 [2026-10-20 06:31:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8851 [2026-10-20 06:31:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3982 [2026-10-20 06:31:35][INFO][training.pretrain_engine] [Ph2|Step 297] loss=6.6797 | ppl=796.09 | aux=0.02514 | lr=2.02e-04 | gnorm=1.781 | tok/s=29,311 | MFU=7.0% | step=143094ms | tokens=0.319B | mem=4.2/32.7GB | cost=$155.89/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 01:39 (139.1h) | Phase3: SKIPPED | SFT: Oct 26 06:23 (143.9h) | DPO (Final): Oct 26 06:57 (144.4h) [2026-10-20 06:31:38][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step297 | Tokens: 0.319B | Spent: $155.90 | Remaining: $244.10 [2026-10-20 06:31:38][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step297 [2026-10-20 06:31:38][INFO][training.pretrain_engine] Starting global step 298 (accumulating 32 micro-batches)... [2026-10-20 06:31:38][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 06:31:38][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 06:31:38][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 06:31:38][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 06:31:38][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step297 to Phase-Technologies/forge-500m-pretrain (path: phase2_step297)... [2026-10-20 06:31:38][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 06:31:47][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 06:31:47][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step297 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 06:31:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8916 [2026-10-20 06:31:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8387 [2026-10-20 06:31:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0226 [2026-10-20 06:31:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7541 [2026-10-20 06:32:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8220 [2026-10-20 06:32:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2818 [2026-10-20 06:32:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1609 [2026-10-20 06:32:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6167 [2026-10-20 06:32:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9757 [2026-10-20 06:32:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8925 [2026-10-20 06:32:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6193 [2026-10-20 06:32:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1231 [2026-10-20 06:32:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9461 [2026-10-20 06:32:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6551 [2026-10-20 06:32:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3323 [2026-10-20 06:32:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2548 [2026-10-20 06:33:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9556 [2026-10-20 06:33:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5227 [2026-10-20 06:33:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6550 [2026-10-20 06:33:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5338 [2026-10-20 06:33:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7051 [2026-10-20 06:33:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7448 [2026-10-20 06:33:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8622 [2026-10-20 06:33:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9498 [2026-10-20 06:33:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4516 [2026-10-20 06:33:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5758 [2026-10-20 06:33:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7696 [2026-10-20 06:33:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6208 [2026-10-20 06:33:57][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,578 | MFU: 1.8% [2026-10-20 06:34:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3227 [2026-10-20 06:34:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9604 [2026-10-20 06:34:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5367 [2026-10-20 06:34:01][INFO][training.pretrain_engine] [Ph2|Step 298] loss=6.6953 | ppl=808.58 | aux=0.02516 | lr=2.03e-04 | gnorm=1.211 | tok/s=29,311 | MFU=7.0% | step=143095ms | tokens=0.320B | mem=4.2/32.7GB | cost=$156.43/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 01:41 (139.1h) | Phase3: SKIPPED | SFT: Oct 26 06:25 (143.9h) | DPO (Final): Oct 26 06:59 (144.4h) [2026-10-20 06:34:01][INFO][training.pretrain_engine] Starting global step 299 (accumulating 32 micro-batches)... [2026-10-20 06:34:01][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 06:34:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6266 [2026-10-20 06:34:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7149 [2026-10-20 06:34:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6072 [2026-10-20 06:34:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9058 [2026-10-20 06:34:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7233 [2026-10-20 06:34:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.6489 [2026-10-20 06:34:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0880 [2026-10-20 06:34:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9490 [2026-10-20 06:34:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9585 [2026-10-20 06:34:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9299 [2026-10-20 06:34:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6260 [2026-10-20 06:34:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3631 [2026-10-20 06:34:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5504 [2026-10-20 06:35:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5182 [2026-10-20 06:35:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7160 [2026-10-20 06:35:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5868 [2026-10-20 06:35:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3352 [2026-10-20 06:35:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5483 [2026-10-20 06:35:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6900 [2026-10-20 06:35:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3904 [2026-10-20 06:35:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6848 [2026-10-20 06:35:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5991 [2026-10-20 06:35:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6824 [2026-10-20 06:35:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8878 [2026-10-20 06:35:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6890 [2026-10-20 06:36:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0722 [2026-10-20 06:36:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3662 [2026-10-20 06:36:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7659 [2026-10-20 06:36:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4373 [2026-10-20 06:36:19][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,601 | MFU: 1.8% [2026-10-20 06:36:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4803 [2026-10-20 06:36:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3695 [2026-10-20 06:36:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9323 [2026-10-20 06:36:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2956 [2026-10-20 06:36:24][INFO][training.pretrain_engine] [Ph2|Step 299] loss=6.6662 | ppl=785.43 | aux=0.02515 | lr=2.03e-04 | gnorm=1.195 | tok/s=29,310 | MFU=7.0% | step=143100ms | tokens=0.321B | mem=4.2/32.7GB | cost=$156.95/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 01:43 (139.1h) | Phase3: SKIPPED | SFT: Oct 26 06:27 (143.9h) | DPO (Final): Oct 26 07:01 (144.4h) [2026-10-20 06:36:24][INFO][training.pretrain_engine] Starting global step 300 (accumulating 32 micro-batches)... [2026-10-20 06:36:24][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 06:36:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5567 [2026-10-20 06:36:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9593 [2026-10-20 06:36:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0361 [2026-10-20 06:36:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5198 [2026-10-20 06:37:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8996 [2026-10-20 06:37:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1638 [2026-10-20 06:37:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8664 [2026-10-20 06:37:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1828 [2026-10-20 06:37:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0769 [2026-10-20 06:37:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9505 [2026-10-20 06:37:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5987 [2026-10-20 06:37:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.2724 [2026-10-20 06:37:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3112 [2026-10-20 06:37:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3947 [2026-10-20 06:37:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3540 [2026-10-20 06:37:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9465 [2026-10-20 06:37:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3679 [2026-10-20 06:37:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3935 [2026-10-20 06:37:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6509 [2026-10-20 06:37:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5886 [2026-10-20 06:38:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3149 [2026-10-20 06:38:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5285 [2026-10-20 06:38:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6218 [2026-10-20 06:38:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5268 [2026-10-20 06:38:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8685 [2026-10-20 06:38:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4020 [2026-10-20 06:38:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1603 [2026-10-20 06:38:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9899 [2026-10-20 06:38:43][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,584 | MFU: 1.8% [2026-10-20 06:38:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8195 [2026-10-20 06:38:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5769 [2026-10-20 06:38:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4993 [2026-10-20 06:38:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8490 [2026-10-20 06:38:48][INFO][training.pretrain_engine] [Ph2|Step 300] loss=6.6621 | ppl=782.21 | aux=0.02514 | lr=2.04e-04 | gnorm=1.328 | tok/s=29,309 | MFU=7.0% | step=143104ms | tokens=0.322B | mem=4.2/32.7GB | cost=$157.47/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 01:45 (139.1h) | Phase3: SKIPPED | SFT: Oct 26 06:29 (143.8h) | DPO (Final): Oct 26 07:03 (144.4h) [2026-10-20 06:38:48][INFO][training.pretrain_engine] Starting global step 301 (accumulating 32 micro-batches)... [2026-10-20 06:38:48][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 06:39:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6333 [2026-10-20 06:39:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6299 [2026-10-20 06:39:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6371 [2026-10-20 06:39:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4366 [2026-10-20 06:39:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0147 [2026-10-20 06:39:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0923 [2026-10-20 06:39:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2544 [2026-10-20 06:39:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1222 [2026-10-20 06:39:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1020 [2026-10-20 06:39:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6646 [2026-10-20 06:39:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.2941 [2026-10-20 06:39:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4099 [2026-10-20 06:39:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6553 [2026-10-20 06:39:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7827 [2026-10-20 06:39:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8316 [2026-10-20 06:39:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5356 [2026-10-20 06:40:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3770 [2026-10-20 06:40:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5095 [2026-10-20 06:40:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.5925 [2026-10-20 06:40:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9795 [2026-10-20 06:40:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5711 [2026-10-20 06:40:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4340 [2026-10-20 06:40:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5624 [2026-10-20 06:40:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2652 [2026-10-20 06:40:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6354 [2026-10-20 06:40:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8830 [2026-10-20 06:40:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6594 [2026-10-20 06:40:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1916 [2026-10-20 06:41:06][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,577 | MFU: 1.8% [2026-10-20 06:41:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5015 [2026-10-20 06:41:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8666 [2026-10-20 06:41:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5236 [2026-10-20 06:41:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4363 [2026-10-20 06:41:11][INFO][training.pretrain_engine] [Ph2|Step 301] loss=6.5807 | ppl=721.07 | aux=0.02514 | lr=2.04e-04 | gnorm=1.070 | tok/s=29,310 | MFU=7.0% | step=143104ms | tokens=0.323B | mem=4.2/32.7GB | cost=$158.00/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 01:47 (139.1h) | Phase3: SKIPPED | SFT: Oct 26 06:31 (143.8h) | DPO (Final): Oct 26 07:05 (144.4h) [2026-10-20 06:41:11][INFO][training.pretrain_engine] Starting global step 302 (accumulating 32 micro-batches)... [2026-10-20 06:41:11][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 06:41:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4913 [2026-10-20 06:41:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6015 [2026-10-20 06:41:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5847 [2026-10-20 06:41:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6060 [2026-10-20 06:41:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4889 [2026-10-20 06:41:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5620 [2026-10-20 06:41:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7501 [2026-10-20 06:41:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9110 [2026-10-20 06:42:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7400 [2026-10-20 06:42:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6313 [2026-10-20 06:42:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0413 [2026-10-20 06:42:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6049 [2026-10-20 06:42:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7973 [2026-10-20 06:42:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8272 [2026-10-20 06:42:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0672 [2026-10-20 06:42:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9752 [2026-10-20 06:42:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1480 [2026-10-20 06:42:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0842 [2026-10-20 06:42:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.3537 [2026-10-20 06:42:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0331 [2026-10-20 06:42:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6469 [2026-10-20 06:42:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4337 [2026-10-20 06:42:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1565 [2026-10-20 06:42:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3818 [2026-10-20 06:43:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8590 [2026-10-20 06:43:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6687 [2026-10-20 06:43:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7852 [2026-10-20 06:43:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3410 [2026-10-20 06:43:29][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,593 | MFU: 1.8% [2026-10-20 06:43:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1565 [2026-10-20 06:43:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0047 [2026-10-20 06:43:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5213 [2026-10-20 06:43:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0945 [2026-10-20 06:43:34][INFO][training.pretrain_engine] [Ph2|Step 302] loss=6.7005 | ppl=812.85 | aux=0.02515 | lr=2.05e-04 | gnorm=1.078 | tok/s=29,309 | MFU=7.0% | step=143105ms | tokens=0.324B | mem=4.2/32.7GB | cost=$158.52/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 01:49 (139.1h) | Phase3: SKIPPED | SFT: Oct 26 06:33 (143.8h) | DPO (Final): Oct 26 07:07 (144.4h) [2026-10-20 06:43:34][INFO][training.pretrain_engine] Starting global step 303 (accumulating 32 micro-batches)... [2026-10-20 06:43:34][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 06:43:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3484 [2026-10-20 06:43:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8433 [2026-10-20 06:43:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4579 [2026-10-20 06:43:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7481 [2026-10-20 06:44:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7547 [2026-10-20 06:44:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6610 [2026-10-20 06:44:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7912 [2026-10-20 06:44:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1752 [2026-10-20 06:44:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5321 [2026-10-20 06:44:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6518 [2026-10-20 06:44:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1981 [2026-10-20 06:44:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5380 [2026-10-20 06:44:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4234 [2026-10-20 06:44:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8893 [2026-10-20 06:44:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4413 [2026-10-20 06:44:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6152 [2026-10-20 06:45:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3648 [2026-10-20 06:45:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7329 [2026-10-20 06:45:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5511 [2026-10-20 06:45:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3207 [2026-10-20 06:45:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9328 [2026-10-20 06:45:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1024 [2026-10-20 06:45:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6047 [2026-10-20 06:45:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5613 [2026-10-20 06:45:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7122 [2026-10-20 06:45:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0738 [2026-10-20 06:45:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8139 [2026-10-20 06:45:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.4433 [2026-10-20 06:45:52][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,601 | MFU: 1.8% [2026-10-20 06:45:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8184 [2026-10-20 06:45:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.1303 [2026-10-20 06:45:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7840 [2026-10-20 06:45:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3058 [2026-10-20 06:45:57][INFO][training.pretrain_engine] [Ph2|Step 303] loss=6.7329 | ppl=839.57 | aux=0.02513 | lr=2.06e-04 | gnorm=1.188 | tok/s=29,310 | MFU=7.0% | step=143102ms | tokens=0.325B | mem=4.2/32.7GB | cost=$159.04/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 01:50 (139.1h) | Phase3: SKIPPED | SFT: Oct 26 06:34 (143.8h) | DPO (Final): Oct 26 07:09 (144.4h) [2026-10-20 06:45:57][INFO][training.pretrain_engine] Starting global step 304 (accumulating 32 micro-batches)... [2026-10-20 06:45:57][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 06:46:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1597 [2026-10-20 06:46:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8832 [2026-10-20 06:46:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1430 [2026-10-20 06:46:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1164 [2026-10-20 06:46:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3780 [2026-10-20 06:46:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5465 [2026-10-20 06:46:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7677 [2026-10-20 06:46:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6851 [2026-10-20 06:46:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1571 [2026-10-20 06:46:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1821 [2026-10-20 06:46:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7395 [2026-10-20 06:46:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5459 [2026-10-20 06:47:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9983 [2026-10-20 06:47:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1921 [2026-10-20 06:47:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4736 [2026-10-20 06:47:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8505 [2026-10-20 06:47:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2545 [2026-10-20 06:47:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8030 [2026-10-20 06:47:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7814 [2026-10-20 06:47:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5916 [2026-10-20 06:47:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6878 [2026-10-20 06:47:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5276 [2026-10-20 06:47:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4362 [2026-10-20 06:47:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5769 [2026-10-20 06:48:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.4770 [2026-10-20 06:48:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6329 [2026-10-20 06:48:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.5685 [2026-10-20 06:48:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7809 [2026-10-20 06:48:15][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,600 | MFU: 1.8% [2026-10-20 06:48:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6306 [2026-10-20 06:48:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4742 [2026-10-20 06:48:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3885 [2026-10-20 06:48:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6458 [2026-10-20 06:48:20][INFO][training.pretrain_engine] [Ph2|Step 304] loss=6.5999 | ppl=735.02 | aux=0.02514 | lr=2.06e-04 | gnorm=1.344 | tok/s=29,310 | MFU=7.0% | step=143104ms | tokens=0.326B | mem=4.2/32.7GB | cost=$159.57/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 01:52 (139.1h) | Phase3: SKIPPED | SFT: Oct 26 06:36 (143.8h) | DPO (Final): Oct 26 07:10 (144.4h) [2026-10-20 06:48:20][INFO][training.pretrain_engine] Starting global step 305 (accumulating 32 micro-batches)... [2026-10-20 06:48:20][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 06:48:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3647 [2026-10-20 06:48:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4254 [2026-10-20 06:48:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4239 [2026-10-20 06:48:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3872 [2026-10-20 06:48:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6777 [2026-10-20 06:48:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2545 [2026-10-20 06:48:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8023 [2026-10-20 06:48:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8014 [2026-10-20 06:49:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0869 [2026-10-20 06:49:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5201 [2026-10-20 06:49:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1929 [2026-10-20 06:49:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3665 [2026-10-20 06:49:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2592 [2026-10-20 06:49:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0836 [2026-10-20 06:49:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9099 [2026-10-20 06:49:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9153 [2026-10-20 06:49:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.6048 [2026-10-20 06:49:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2314 [2026-10-20 06:49:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.5282 [2026-10-20 06:49:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.5177 [2026-10-20 06:50:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3745 [2026-10-20 06:50:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9610 [2026-10-20 06:50:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9571 [2026-10-20 06:50:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9538 [2026-10-20 06:50:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7766 [2026-10-20 06:50:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4652 [2026-10-20 06:50:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3924 [2026-10-20 06:50:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5948 [2026-10-20 06:50:38][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,600 | MFU: 1.8% [2026-10-20 06:50:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5362 [2026-10-20 06:50:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5748 [2026-10-20 06:50:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5228 [2026-10-20 06:50:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9068 [2026-10-20 06:50:43][INFO][training.pretrain_engine] [Ph2|Step 305] loss=6.7390 | ppl=844.69 | aux=0.02513 | lr=2.07e-04 | gnorm=0.836 | tok/s=29,310 | MFU=7.0% | step=143104ms | tokens=0.327B | mem=4.2/32.7GB | cost=$160.09/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 01:54 (139.1h) | Phase3: SKIPPED | SFT: Oct 26 06:38 (143.8h) | DPO (Final): Oct 26 07:12 (144.4h) [2026-10-20 06:50:43][INFO][training.pretrain_engine] Starting global step 306 (accumulating 32 micro-batches)... [2026-10-20 06:50:43][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 06:51:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9727 [2026-10-20 06:51:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1437 [2026-10-20 06:51:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5204 [2026-10-20 06:51:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2749 [2026-10-20 06:51:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4614 [2026-10-20 06:51:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0976 [2026-10-20 06:51:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5162 [2026-10-20 06:51:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2337 [2026-10-20 06:51:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3765 [2026-10-20 06:51:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3744 [2026-10-20 06:51:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5620 [2026-10-20 06:51:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4740 [2026-10-20 06:51:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6241 [2026-10-20 06:51:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6937 [2026-10-20 06:51:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9426 [2026-10-20 06:51:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4712 [2026-10-20 06:52:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4946 [2026-10-20 06:52:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7269 [2026-10-20 06:52:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4245 [2026-10-20 06:52:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9566 [2026-10-20 06:52:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6571 [2026-10-20 06:52:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4221 [2026-10-20 06:52:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6389 [2026-10-20 06:52:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.3052 [2026-10-20 06:52:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1185 [2026-10-20 06:52:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1380 [2026-10-20 06:52:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9321 [2026-10-20 06:52:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3982 [2026-10-20 06:53:01][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,592 | MFU: 1.8% [2026-10-20 06:53:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5083 [2026-10-20 06:53:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3472 [2026-10-20 06:53:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3702 [2026-10-20 06:53:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4137 [2026-10-20 06:53:06][INFO][training.pretrain_engine] [Ph2|Step 306] loss=6.4929 | ppl=660.46 | aux=0.02515 | lr=2.07e-04 | gnorm=1.430 | tok/s=29,309 | MFU=7.0% | step=143105ms | tokens=0.328B | mem=4.2/32.7GB | cost=$160.61/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 01:56 (139.1h) | Phase3: SKIPPED | SFT: Oct 26 06:40 (143.8h) | DPO (Final): Oct 26 07:14 (144.4h) [2026-10-20 06:53:09][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step306 | Tokens: 0.328B | Spent: $160.62 | Remaining: $239.38 [2026-10-20 06:53:09][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step306 [2026-10-20 06:53:09][INFO][training.pretrain_engine] Starting global step 307 (accumulating 32 micro-batches)... [2026-10-20 06:53:09][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 06:53:09][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 06:53:09][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 06:53:09][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 06:53:09][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step306 to Phase-Technologies/forge-500m-pretrain (path: phase2_step306)... [2026-10-20 06:53:09][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 06:53:18][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 06:53:18][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step306 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 06:53:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5802 [2026-10-20 06:53:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5951 [2026-10-20 06:53:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8212 [2026-10-20 06:53:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6958 [2026-10-20 06:53:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3465 [2026-10-20 06:53:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6009 [2026-10-20 06:53:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7928 [2026-10-20 06:53:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6271 [2026-10-20 06:53:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6870 [2026-10-20 06:53:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3756 [2026-10-20 06:53:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5621 [2026-10-20 06:54:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5197 [2026-10-20 06:54:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2987 [2026-10-20 06:54:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6231 [2026-10-20 06:54:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5395 [2026-10-20 06:54:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0910 [2026-10-20 06:54:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9438 [2026-10-20 06:54:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9458 [2026-10-20 06:54:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9934 [2026-10-20 06:54:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9525 [2026-10-20 06:54:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8726 [2026-10-20 06:54:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8584 [2026-10-20 06:54:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9952 [2026-10-20 06:54:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9795 [2026-10-20 06:55:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5419 [2026-10-20 06:55:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5877 [2026-10-20 06:55:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4041 [2026-10-20 06:55:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4821 [2026-10-20 06:55:27][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,585 | MFU: 1.8% [2026-10-20 06:55:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4892 [2026-10-20 06:55:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4136 [2026-10-20 06:55:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5912 [2026-10-20 06:55:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6081 [2026-10-20 06:55:32][INFO][training.pretrain_engine] [Ph2|Step 307] loss=6.6391 | ppl=764.43 | aux=0.02514 | lr=2.08e-04 | gnorm=1.211 | tok/s=29,310 | MFU=7.0% | step=143102ms | tokens=0.329B | mem=4.2/32.7GB | cost=$161.14/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 01:57 (139.0h) | Phase3: SKIPPED | SFT: Oct 26 06:42 (143.8h) | DPO (Final): Oct 26 07:16 (144.3h) [2026-10-20 06:55:32][INFO][training.pretrain_engine] Starting global step 308 (accumulating 32 micro-batches)... [2026-10-20 06:55:32][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 06:55:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8831 [2026-10-20 06:55:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8614 [2026-10-20 06:55:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2606 [2026-10-20 06:55:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5612 [2026-10-20 06:56:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0572 [2026-10-20 06:56:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1386 [2026-10-20 06:56:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0578 [2026-10-20 06:56:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9442 [2026-10-20 06:56:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1207 [2026-10-20 06:56:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0166 [2026-10-20 06:56:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4160 [2026-10-20 06:56:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0407 [2026-10-20 06:56:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1105 [2026-10-20 06:56:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2477 [2026-10-20 06:56:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.3473 [2026-10-20 06:56:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.5207 [2026-10-20 06:57:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8062 [2026-10-20 06:57:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5830 [2026-10-20 06:57:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5496 [2026-10-20 06:57:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6066 [2026-10-20 06:57:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2363 [2026-10-20 06:57:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7620 [2026-10-20 06:57:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2467 [2026-10-20 06:57:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8253 [2026-10-20 06:57:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5348 [2026-10-20 06:57:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1750 [2026-10-20 06:57:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8369 [2026-10-20 06:57:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3801 [2026-10-20 06:57:50][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,546 | MFU: 1.8% [2026-10-20 06:57:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7969 [2026-10-20 06:57:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9394 [2026-10-20 06:57:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.1449 [2026-10-20 06:57:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8112 [2026-10-20 06:57:55][INFO][training.pretrain_engine] [Ph2|Step 308] loss=6.6046 | ppl=738.48 | aux=0.02513 | lr=2.08e-04 | gnorm=1.211 | tok/s=29,309 | MFU=7.0% | step=143104ms | tokens=0.330B | mem=4.2/32.7GB | cost=$161.67/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 01:59 (139.0h) | Phase3: SKIPPED | SFT: Oct 26 06:44 (143.8h) | DPO (Final): Oct 26 07:18 (144.3h) [2026-10-20 06:57:55][INFO][training.pretrain_engine] Starting global step 309 (accumulating 32 micro-batches)... [2026-10-20 06:57:55][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 06:58:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5617 [2026-10-20 06:58:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9588 [2026-10-20 06:58:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3889 [2026-10-20 06:58:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4713 [2026-10-20 06:58:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7665 [2026-10-20 06:58:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7711 [2026-10-20 06:58:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1648 [2026-10-20 06:58:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8210 [2026-10-20 06:58:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1483 [2026-10-20 06:58:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5439 [2026-10-20 06:58:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6382 [2026-10-20 06:58:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6392 [2026-10-20 06:59:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6197 [2026-10-20 06:59:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5697 [2026-10-20 06:59:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4409 [2026-10-20 06:59:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5952 [2026-10-20 06:59:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7647 [2026-10-20 06:59:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4747 [2026-10-20 06:59:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1601 [2026-10-20 06:59:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8156 [2026-10-20 06:59:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.3496 [2026-10-20 06:59:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5708 [2026-10-20 06:59:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6395 [2026-10-20 06:59:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6999 [2026-10-20 07:00:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7951 [2026-10-20 07:00:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0239 [2026-10-20 07:00:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8360 [2026-10-20 07:00:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.4068 [2026-10-20 07:00:14][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,539 | MFU: 1.8% [2026-10-20 07:00:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9035 [2026-10-20 07:00:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9917 [2026-10-20 07:00:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1325 [2026-10-20 07:00:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0368 [2026-10-20 07:00:19][INFO][training.pretrain_engine] [Ph2|Step 309] loss=6.6716 | ppl=789.68 | aux=0.02515 | lr=2.09e-04 | gnorm=1.000 | tok/s=29,308 | MFU=7.0% | step=143112ms | tokens=0.331B | mem=4.2/32.7GB | cost=$162.19/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 02:02 (139.0h) | Phase3: SKIPPED | SFT: Oct 26 06:46 (143.8h) | DPO (Final): Oct 26 07:20 (144.3h) [2026-10-20 07:00:19][INFO][training.pretrain_engine] Starting global step 310 (accumulating 32 micro-batches)... [2026-10-20 07:00:19][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 07:00:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8374 [2026-10-20 07:00:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7224 [2026-10-20 07:00:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5347 [2026-10-20 07:00:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6438 [2026-10-20 07:00:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7432 [2026-10-20 07:00:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8071 [2026-10-20 07:00:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3622 [2026-10-20 07:00:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5865 [2026-10-20 07:01:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7014 [2026-10-20 07:01:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6001 [2026-10-20 07:01:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8708 [2026-10-20 07:01:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2615 [2026-10-20 07:01:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2891 [2026-10-20 07:01:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6954 [2026-10-20 07:01:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6351 [2026-10-20 07:01:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0247 [2026-10-20 07:01:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6444 [2026-10-20 07:01:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2072 [2026-10-20 07:01:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3281 [2026-10-20 07:01:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3425 [2026-10-20 07:02:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8409 [2026-10-20 07:02:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3247 [2026-10-20 07:02:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2399 [2026-10-20 07:02:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2292 [2026-10-20 07:02:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7127 [2026-10-20 07:02:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6137 [2026-10-20 07:02:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1471 [2026-10-20 07:02:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4805 [2026-10-20 07:02:38][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,546 | MFU: 1.8% [2026-10-20 07:02:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4857 [2026-10-20 07:02:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6412 [2026-10-20 07:02:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9565 [2026-10-20 07:02:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9844 [2026-10-20 07:02:42][INFO][training.pretrain_engine] [Ph2|Step 310] loss=6.5373 | ppl=690.43 | aux=0.02513 | lr=2.09e-04 | gnorm=1.164 | tok/s=29,307 | MFU=7.0% | step=143117ms | tokens=0.332B | mem=4.2/32.7GB | cost=$162.72/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 02:04 (139.0h) | Phase3: SKIPPED | SFT: Oct 26 06:48 (143.8h) | DPO (Final): Oct 26 07:22 (144.3h) [2026-10-20 07:02:42][INFO][training.pretrain_engine] Starting global step 311 (accumulating 32 micro-batches)... [2026-10-20 07:02:42][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 07:03:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8420 [2026-10-20 07:03:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0777 [2026-10-20 07:03:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6965 [2026-10-20 07:03:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4947 [2026-10-20 07:03:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4587 [2026-10-20 07:03:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4347 [2026-10-20 07:03:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6577 [2026-10-20 07:03:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3374 [2026-10-20 07:03:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3979 [2026-10-20 07:03:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3393 [2026-10-20 07:03:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2961 [2026-10-20 07:03:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1907 [2026-10-20 07:03:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0431 [2026-10-20 07:03:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7481 [2026-10-20 07:03:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9449 [2026-10-20 07:03:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5423 [2026-10-20 07:04:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7285 [2026-10-20 07:04:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1765 [2026-10-20 07:04:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4688 [2026-10-20 07:04:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6189 [2026-10-20 07:04:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9444 [2026-10-20 07:04:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0244 [2026-10-20 07:04:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2934 [2026-10-20 07:04:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1439 [2026-10-20 07:04:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4559 [2026-10-20 07:04:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3208 [2026-10-20 07:04:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9219 [2026-10-20 07:04:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1625 [2026-10-20 07:05:01][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,553 | MFU: 1.8% [2026-10-20 07:05:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5904 [2026-10-20 07:05:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9555 [2026-10-20 07:05:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0074 [2026-10-20 07:05:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7298 [2026-10-20 07:05:06][INFO][training.pretrain_engine] [Ph2|Step 311] loss=6.5122 | ppl=673.27 | aux=0.02512 | lr=2.10e-04 | gnorm=1.039 | tok/s=29,306 | MFU=7.0% | step=143120ms | tokens=0.333B | mem=4.2/32.7GB | cost=$163.24/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 02:06 (139.0h) | Phase3: SKIPPED | SFT: Oct 26 06:50 (143.8h) | DPO (Final): Oct 26 07:24 (144.3h) [2026-10-20 07:05:06][INFO][training.pretrain_engine] Starting global step 312 (accumulating 32 micro-batches)... [2026-10-20 07:05:06][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 07:05:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8845 [2026-10-20 07:05:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2790 [2026-10-20 07:05:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0768 [2026-10-20 07:05:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3889 [2026-10-20 07:05:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6565 [2026-10-20 07:05:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2921 [2026-10-20 07:05:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2799 [2026-10-20 07:05:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9539 [2026-10-20 07:05:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8957 [2026-10-20 07:05:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6245 [2026-10-20 07:05:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7601 [2026-10-20 07:05:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0051 [2026-10-20 07:06:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3092 [2026-10-20 07:06:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0254 [2026-10-20 07:06:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4458 [2026-10-20 07:06:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3272 [2026-10-20 07:06:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0801 [2026-10-20 07:06:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4740 [2026-10-20 07:06:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2467 [2026-10-20 07:06:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5680 [2026-10-20 07:06:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8792 [2026-10-20 07:06:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9367 [2026-10-20 07:06:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8703 [2026-10-20 07:06:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0178 [2026-10-20 07:07:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1934 [2026-10-20 07:07:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2431 [2026-10-20 07:07:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5376 [2026-10-20 07:07:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1606 [2026-10-20 07:07:24][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,588 | MFU: 1.8% [2026-10-20 07:07:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3856 [2026-10-20 07:07:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1780 [2026-10-20 07:07:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5401 [2026-10-20 07:07:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3116 [2026-10-20 07:07:29][INFO][training.pretrain_engine] [Ph2|Step 312] loss=6.4963 | ppl=662.71 | aux=0.02515 | lr=2.11e-04 | gnorm=1.367 | tok/s=29,305 | MFU=7.0% | step=143124ms | tokens=0.334B | mem=4.2/32.7GB | cost=$163.77/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 02:08 (139.0h) | Phase3: SKIPPED | SFT: Oct 26 06:52 (143.7h) | DPO (Final): Oct 26 07:26 (144.3h) [2026-10-20 07:07:29][INFO][training.pretrain_engine] Starting global step 313 (accumulating 32 micro-batches)... [2026-10-20 07:07:29][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 07:07:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3822 [2026-10-20 07:07:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.5254 [2026-10-20 07:07:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0866 [2026-10-20 07:07:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4081 [2026-10-20 07:08:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6495 [2026-10-20 07:08:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2883 [2026-10-20 07:08:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5977 [2026-10-20 07:08:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2703 [2026-10-20 07:08:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1149 [2026-10-20 07:08:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3581 [2026-10-20 07:08:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3179 [2026-10-20 07:08:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7351 [2026-10-20 07:08:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6558 [2026-10-20 07:08:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8029 [2026-10-20 07:08:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1645 [2026-10-20 07:08:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9868 [2026-10-20 07:08:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7698 [2026-10-20 07:08:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3018 [2026-10-20 07:08:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0759 [2026-10-20 07:08:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5926 [2026-10-20 07:09:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7925 [2026-10-20 07:09:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0451 [2026-10-20 07:09:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0272 [2026-10-20 07:09:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9947 [2026-10-20 07:09:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8353 [2026-10-20 07:09:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3683 [2026-10-20 07:09:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8566 [2026-10-20 07:09:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7927 [2026-10-20 07:09:48][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,548 | MFU: 1.8% [2026-10-20 07:09:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8276 [2026-10-20 07:09:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1023 [2026-10-20 07:09:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9562 [2026-10-20 07:09:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3027 [2026-10-20 07:09:53][INFO][training.pretrain_engine] [Ph2|Step 313] loss=6.6477 | ppl=771.00 | aux=0.02514 | lr=2.11e-04 | gnorm=0.832 | tok/s=29,304 | MFU=7.0% | step=143130ms | tokens=0.336B | mem=4.2/32.7GB | cost=$164.29/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 02:10 (139.0h) | Phase3: SKIPPED | SFT: Oct 26 06:54 (143.7h) | DPO (Final): Oct 26 07:28 (144.3h) [2026-10-20 07:09:53][INFO][training.pretrain_engine] Starting global step 314 (accumulating 32 micro-batches)... [2026-10-20 07:09:53][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 07:10:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2141 [2026-10-20 07:10:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6288 [2026-10-20 07:10:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0336 [2026-10-20 07:10:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1827 [2026-10-20 07:10:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4547 [2026-10-20 07:10:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6178 [2026-10-20 07:10:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3337 [2026-10-20 07:10:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6438 [2026-10-20 07:10:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7880 [2026-10-20 07:10:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1064 [2026-10-20 07:10:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4041 [2026-10-20 07:10:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2852 [2026-10-20 07:11:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4887 [2026-10-20 07:11:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0282 [2026-10-20 07:11:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3558 [2026-10-20 07:11:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4314 [2026-10-20 07:11:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4246 [2026-10-20 07:11:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3095 [2026-10-20 07:11:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1125 [2026-10-20 07:11:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6898 [2026-10-20 07:11:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4651 [2026-10-20 07:11:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9195 [2026-10-20 07:11:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5521 [2026-10-20 07:11:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1150 [2026-10-20 07:11:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0500 [2026-10-20 07:11:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6572 [2026-10-20 07:11:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.4286 [2026-10-20 07:11:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3340 [2026-10-20 07:12:12][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,544 | MFU: 1.8% [2026-10-20 07:12:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2521 [2026-10-20 07:12:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3121 [2026-10-20 07:12:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4521 [2026-10-20 07:12:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7515 [2026-10-20 07:12:16][INFO][training.pretrain_engine] [Ph2|Step 314] loss=6.6959 | ppl=809.09 | aux=0.02514 | lr=2.12e-04 | gnorm=0.816 | tok/s=29,303 | MFU=7.0% | step=143136ms | tokens=0.337B | mem=4.2/32.7GB | cost=$164.82/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 02:12 (139.0h) | Phase3: SKIPPED | SFT: Oct 26 06:56 (143.7h) | DPO (Final): Oct 26 07:30 (144.3h) [2026-10-20 07:12:16][INFO][training.pretrain_engine] Starting global step 315 (accumulating 32 micro-batches)... [2026-10-20 07:12:16][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 07:12:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7148 [2026-10-20 07:12:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2061 [2026-10-20 07:12:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0775 [2026-10-20 07:12:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7478 [2026-10-20 07:12:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4239 [2026-10-20 07:12:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5400 [2026-10-20 07:12:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1867 [2026-10-20 07:12:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5018 [2026-10-20 07:13:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3705 [2026-10-20 07:13:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7241 [2026-10-20 07:13:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5689 [2026-10-20 07:13:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3477 [2026-10-20 07:13:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4029 [2026-10-20 07:13:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6501 [2026-10-20 07:13:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5071 [2026-10-20 07:13:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6696 [2026-10-20 07:13:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7187 [2026-10-20 07:13:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6841 [2026-10-20 07:13:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.5624 [2026-10-20 07:13:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6649 [2026-10-20 07:14:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4076 [2026-10-20 07:14:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3297 [2026-10-20 07:14:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9383 [2026-10-20 07:14:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1186 [2026-10-20 07:14:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6700 [2026-10-20 07:14:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0699 [2026-10-20 07:14:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6268 [2026-10-20 07:14:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4915 [2026-10-20 07:14:35][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,529 | MFU: 1.8% [2026-10-20 07:14:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5713 [2026-10-20 07:14:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6862 [2026-10-20 07:14:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6034 [2026-10-20 07:14:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3331 [2026-10-20 07:14:40][INFO][training.pretrain_engine] [Ph2|Step 315] loss=6.5745 | ppl=716.61 | aux=0.02515 | lr=2.12e-04 | gnorm=1.078 | tok/s=29,302 | MFU=7.0% | step=143140ms | tokens=0.338B | mem=4.2/32.7GB | cost=$165.34/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 02:14 (139.0h) | Phase3: SKIPPED | SFT: Oct 26 06:58 (143.7h) | DPO (Final): Oct 26 07:32 (144.3h) [2026-10-20 07:14:43][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step315 | Tokens: 0.338B | Spent: $165.35 | Remaining: $234.65 [2026-10-20 07:14:43][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step315 [2026-10-20 07:14:43][INFO][training.pretrain_engine] Starting global step 316 (accumulating 32 micro-batches)... [2026-10-20 07:14:43][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 07:14:43][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 07:14:43][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 07:14:43][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 07:14:43][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step315 to Phase-Technologies/forge-500m-pretrain (path: phase2_step315)... [2026-10-20 07:14:43][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 07:14:53][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 07:14:53][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step315 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 07:14:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4199 [2026-10-20 07:14:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6384 [2026-10-20 07:14:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2889 [2026-10-20 07:15:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4191 [2026-10-20 07:15:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2243 [2026-10-20 07:15:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9484 [2026-10-20 07:15:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7572 [2026-10-20 07:15:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3267 [2026-10-20 07:15:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7270 [2026-10-20 07:15:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7292 [2026-10-20 07:15:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8376 [2026-10-20 07:15:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6916 [2026-10-20 07:15:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4932 [2026-10-20 07:15:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8377 [2026-10-20 07:15:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6377 [2026-10-20 07:15:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7206 [2026-10-20 07:16:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4033 [2026-10-20 07:16:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6692 [2026-10-20 07:16:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2795 [2026-10-20 07:16:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5019 [2026-10-20 07:16:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0749 [2026-10-20 07:16:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3500 [2026-10-20 07:16:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4292 [2026-10-20 07:16:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5852 [2026-10-20 07:16:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1694 [2026-10-20 07:16:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6360 [2026-10-20 07:16:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4001 [2026-10-20 07:16:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7638 [2026-10-20 07:17:03][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,521 | MFU: 1.8% [2026-10-20 07:17:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3206 [2026-10-20 07:17:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.1753 [2026-10-20 07:17:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.1790 [2026-10-20 07:17:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2044 [2026-10-20 07:17:07][INFO][training.pretrain_engine] [Ph2|Step 316] loss=6.6067 | ppl=740.07 | aux=0.02512 | lr=2.13e-04 | gnorm=1.242 | tok/s=29,300 | MFU=7.0% | step=143149ms | tokens=0.339B | mem=4.2/32.7GB | cost=$165.88/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 02:16 (139.0h) | Phase3: SKIPPED | SFT: Oct 26 07:01 (143.7h) | DPO (Final): Oct 26 07:35 (144.3h) [2026-10-20 07:17:07][INFO][training.pretrain_engine] Starting global step 317 (accumulating 32 micro-batches)... [2026-10-20 07:17:07][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 07:17:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5336 [2026-10-20 07:17:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1851 [2026-10-20 07:17:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5890 [2026-10-20 07:17:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7762 [2026-10-20 07:17:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0224 [2026-10-20 07:17:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2510 [2026-10-20 07:17:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1902 [2026-10-20 07:17:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1408 [2026-10-20 07:18:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3627 [2026-10-20 07:18:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6537 [2026-10-20 07:18:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4226 [2026-10-20 07:18:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5823 [2026-10-20 07:18:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7993 [2026-10-20 07:18:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5879 [2026-10-20 07:18:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7604 [2026-10-20 07:18:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8855 [2026-10-20 07:18:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0424 [2026-10-20 07:18:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7214 [2026-10-20 07:18:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0968 [2026-10-20 07:18:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5252 [2026-10-20 07:18:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7213 [2026-10-20 07:18:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1994 [2026-10-20 07:18:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.5430 [2026-10-20 07:18:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8752 [2026-10-20 07:19:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3274 [2026-10-20 07:19:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3278 [2026-10-20 07:19:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4210 [2026-10-20 07:19:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1747 [2026-10-20 07:19:26][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,540 | MFU: 1.8% [2026-10-20 07:19:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4914 [2026-10-20 07:19:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2747 [2026-10-20 07:19:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1734 [2026-10-20 07:19:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5859 [2026-10-20 07:19:31][INFO][training.pretrain_engine] [Ph2|Step 317] loss=6.5283 | ppl=684.23 | aux=0.02513 | lr=2.13e-04 | gnorm=1.141 | tok/s=29,297 | MFU=7.0% | step=143163ms | tokens=0.340B | mem=4.2/32.7GB | cost=$166.41/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 02:19 (139.0h) | Phase3: SKIPPED | SFT: Oct 26 07:03 (143.7h) | DPO (Final): Oct 26 07:38 (144.3h) [2026-10-20 07:19:31][INFO][training.pretrain_engine] Starting global step 318 (accumulating 32 micro-batches)... [2026-10-20 07:19:31][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 07:19:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3030 [2026-10-20 07:19:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0500 [2026-10-20 07:19:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5801 [2026-10-20 07:19:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6154 [2026-10-20 07:20:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5907 [2026-10-20 07:20:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3207 [2026-10-20 07:20:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4194 [2026-10-20 07:20:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7992 [2026-10-20 07:20:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9254 [2026-10-20 07:20:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0569 [2026-10-20 07:20:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0572 [2026-10-20 07:20:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7751 [2026-10-20 07:20:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2847 [2026-10-20 07:20:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3677 [2026-10-20 07:20:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1533 [2026-10-20 07:20:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3823 [2026-10-20 07:21:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8867 [2026-10-20 07:21:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3574 [2026-10-20 07:21:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4663 [2026-10-20 07:21:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4858 [2026-10-20 07:21:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3891 [2026-10-20 07:21:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4968 [2026-10-20 07:21:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3590 [2026-10-20 07:21:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2756 [2026-10-20 07:21:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6648 [2026-10-20 07:21:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5636 [2026-10-20 07:21:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5604 [2026-10-20 07:21:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3922 [2026-10-20 07:21:50][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,532 | MFU: 1.8% [2026-10-20 07:21:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0960 [2026-10-20 07:21:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2325 [2026-10-20 07:21:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1619 [2026-10-20 07:21:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4397 [2026-10-20 07:21:55][INFO][training.pretrain_engine] [Ph2|Step 318] loss=6.5160 | ppl=675.86 | aux=0.02512 | lr=2.14e-04 | gnorm=0.891 | tok/s=29,295 | MFU=7.0% | step=143173ms | tokens=0.341B | mem=4.2/32.7GB | cost=$166.93/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 02:21 (139.0h) | Phase3: SKIPPED | SFT: Oct 26 07:06 (143.7h) | DPO (Final): Oct 26 07:40 (144.3h) [2026-10-20 07:21:55][INFO][training.pretrain_engine] Starting global step 319 (accumulating 32 micro-batches)... [2026-10-20 07:21:55][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 07:22:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6621 [2026-10-20 07:22:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8120 [2026-10-20 07:22:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6701 [2026-10-20 07:22:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2422 [2026-10-20 07:22:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7831 [2026-10-20 07:22:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1461 [2026-10-20 07:22:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7412 [2026-10-20 07:22:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6722 [2026-10-20 07:22:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5232 [2026-10-20 07:22:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5458 [2026-10-20 07:22:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3809 [2026-10-20 07:22:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5299 [2026-10-20 07:23:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2646 [2026-10-20 07:23:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4807 [2026-10-20 07:23:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6982 [2026-10-20 07:23:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3775 [2026-10-20 07:23:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4280 [2026-10-20 07:23:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5504 [2026-10-20 07:23:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6875 [2026-10-20 07:23:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7075 [2026-10-20 07:23:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2329 [2026-10-20 07:23:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5678 [2026-10-20 07:23:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7458 [2026-10-20 07:23:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5249 [2026-10-20 07:24:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5790 [2026-10-20 07:24:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4833 [2026-10-20 07:24:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4521 [2026-10-20 07:24:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3402 [2026-10-20 07:24:14][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,523 | MFU: 1.8% [2026-10-20 07:24:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5702 [2026-10-20 07:24:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5258 [2026-10-20 07:24:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4135 [2026-10-20 07:24:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.3470 [2026-10-20 07:24:19][INFO][training.pretrain_engine] [Ph2|Step 319] loss=6.6075 | ppl=740.61 | aux=0.02512 | lr=2.15e-04 | gnorm=0.746 | tok/s=29,293 | MFU=7.0% | step=143183ms | tokens=0.342B | mem=4.2/32.7GB | cost=$167.46/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 02:24 (139.0h) | Phase3: SKIPPED | SFT: Oct 26 07:08 (143.7h) | DPO (Final): Oct 26 07:42 (144.3h) [2026-10-20 07:24:19][INFO][training.pretrain_engine] Starting global step 320 (accumulating 32 micro-batches)... [2026-10-20 07:24:19][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 07:24:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8106 [2026-10-20 07:24:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7495 [2026-10-20 07:24:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2861 [2026-10-20 07:24:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2166 [2026-10-20 07:24:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3458 [2026-10-20 07:24:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4296 [2026-10-20 07:24:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9502 [2026-10-20 07:24:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5503 [2026-10-20 07:25:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1691 [2026-10-20 07:25:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8634 [2026-10-20 07:25:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2747 [2026-10-20 07:25:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0384 [2026-10-20 07:25:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5801 [2026-10-20 07:25:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9695 [2026-10-20 07:25:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7553 [2026-10-20 07:25:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6453 [2026-10-20 07:25:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7905 [2026-10-20 07:25:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9104 [2026-10-20 07:25:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8942 [2026-10-20 07:25:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2268 [2026-10-20 07:26:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1942 [2026-10-20 07:26:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4559 [2026-10-20 07:26:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7066 [2026-10-20 07:26:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8265 [2026-10-20 07:26:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4080 [2026-10-20 07:26:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1842 [2026-10-20 07:26:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8838 [2026-10-20 07:26:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3941 [2026-10-20 07:26:38][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,526 | MFU: 1.8% [2026-10-20 07:26:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1029 [2026-10-20 07:26:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2637 [2026-10-20 07:26:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4729 [2026-10-20 07:26:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2022 [2026-10-20 07:26:43][INFO][training.pretrain_engine] [Ph2|Step 320] loss=6.5469 | ppl=697.11 | aux=0.02513 | lr=2.15e-04 | gnorm=1.133 | tok/s=29,291 | MFU=7.0% | step=143197ms | tokens=0.343B | mem=4.2/32.7GB | cost=$167.99/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 02:26 (139.0h) | Phase3: SKIPPED | SFT: Oct 26 07:11 (143.7h) | DPO (Final): Oct 26 07:45 (144.3h) [2026-10-20 07:26:43][INFO][training.pretrain_engine] Starting global step 321 (accumulating 32 micro-batches)... [2026-10-20 07:26:43][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 07:27:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2653 [2026-10-20 07:27:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8716 [2026-10-20 07:27:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9721 [2026-10-20 07:27:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7052 [2026-10-20 07:27:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1167 [2026-10-20 07:27:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5632 [2026-10-20 07:27:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2159 [2026-10-20 07:27:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2498 [2026-10-20 07:27:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4859 [2026-10-20 07:27:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8017 [2026-10-20 07:27:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5796 [2026-10-20 07:27:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5229 [2026-10-20 07:27:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3519 [2026-10-20 07:27:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0402 [2026-10-20 07:27:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3250 [2026-10-20 07:27:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2658 [2026-10-20 07:28:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6167 [2026-10-20 07:28:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4965 [2026-10-20 07:28:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2328 [2026-10-20 07:28:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5383 [2026-10-20 07:28:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0731 [2026-10-20 07:28:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5967 [2026-10-20 07:28:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3921 [2026-10-20 07:28:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9581 [2026-10-20 07:28:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1566 [2026-10-20 07:28:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1386 [2026-10-20 07:28:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3529 [2026-10-20 07:28:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4057 [2026-10-20 07:29:02][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,560 | MFU: 1.8% [2026-10-20 07:29:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4213 [2026-10-20 07:29:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7931 [2026-10-20 07:29:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3302 [2026-10-20 07:29:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3670 [2026-10-20 07:29:08][INFO][training.pretrain_engine] [Ph2|Step 321] loss=6.5154 | ppl=675.49 | aux=0.02513 | lr=2.16e-04 | gnorm=1.258 | tok/s=29,287 | MFU=7.0% | step=143213ms | tokens=0.344B | mem=4.2/32.7GB | cost=$168.51/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 02:29 (139.0h) | Phase3: SKIPPED | SFT: Oct 26 07:14 (143.7h) | DPO (Final): Oct 26 07:48 (144.3h) [2026-10-20 07:29:08][INFO][training.pretrain_engine] Starting global step 322 (accumulating 32 micro-batches)... [2026-10-20 07:29:08][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 07:29:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2615 [2026-10-20 07:29:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4730 [2026-10-20 07:29:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8121 [2026-10-20 07:29:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2811 [2026-10-20 07:29:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4086 [2026-10-20 07:29:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2892 [2026-10-20 07:29:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1582 [2026-10-20 07:29:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4322 [2026-10-20 07:30:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6545 [2026-10-20 07:30:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7003 [2026-10-20 07:30:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6026 [2026-10-20 07:30:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5115 [2026-10-20 07:30:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4909 [2026-10-20 07:30:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1309 [2026-10-20 07:30:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5450 [2026-10-20 07:30:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2117 [2026-10-20 07:30:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.5118 [2026-10-20 07:30:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6566 [2026-10-20 07:30:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8001 [2026-10-20 07:30:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1383 [2026-10-20 07:30:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3344 [2026-10-20 07:30:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6597 [2026-10-20 07:30:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6488 [2026-10-20 07:30:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2346 [2026-10-20 07:31:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.5478 [2026-10-20 07:31:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1869 [2026-10-20 07:31:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1570 [2026-10-20 07:31:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.4066 [2026-10-20 07:31:27][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,518 | MFU: 1.8% [2026-10-20 07:31:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6615 [2026-10-20 07:31:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4664 [2026-10-20 07:31:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5003 [2026-10-20 07:31:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8064 [2026-10-20 07:31:32][INFO][training.pretrain_engine] [Ph2|Step 322] loss=6.5244 | ppl=681.58 | aux=0.02514 | lr=2.16e-04 | gnorm=0.887 | tok/s=29,285 | MFU=7.0% | step=143223ms | tokens=0.345B | mem=4.2/32.7GB | cost=$169.04/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 02:31 (139.0h) | Phase3: SKIPPED | SFT: Oct 26 07:16 (143.7h) | DPO (Final): Oct 26 07:50 (144.3h) [2026-10-20 07:31:32][INFO][training.pretrain_engine] Starting global step 323 (accumulating 32 micro-batches)... [2026-10-20 07:31:32][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 07:31:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0978 [2026-10-20 07:31:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3424 [2026-10-20 07:31:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7790 [2026-10-20 07:31:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3109 [2026-10-20 07:32:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7232 [2026-10-20 07:32:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=8.1130 [2026-10-20 07:32:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0211 [2026-10-20 07:32:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8609 [2026-10-20 07:32:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2690 [2026-10-20 07:32:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2579 [2026-10-20 07:32:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3368 [2026-10-20 07:32:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3972 [2026-10-20 07:32:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5252 [2026-10-20 07:32:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6258 [2026-10-20 07:32:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8246 [2026-10-20 07:32:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4919 [2026-10-20 07:33:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3437 [2026-10-20 07:33:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6419 [2026-10-20 07:33:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2072 [2026-10-20 07:33:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2302 [2026-10-20 07:33:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3977 [2026-10-20 07:33:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7779 [2026-10-20 07:33:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6977 [2026-10-20 07:33:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0499 [2026-10-20 07:33:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7513 [2026-10-20 07:33:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9063 [2026-10-20 07:33:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4388 [2026-10-20 07:33:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2336 [2026-10-20 07:33:51][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,519 | MFU: 1.8% [2026-10-20 07:33:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4959 [2026-10-20 07:33:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3483 [2026-10-20 07:33:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.1542 [2026-10-20 07:33:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6168 [2026-10-20 07:33:56][INFO][training.pretrain_engine] [Ph2|Step 323] loss=6.6188 | ppl=749.03 | aux=0.02513 | lr=2.17e-04 | gnorm=0.930 | tok/s=29,282 | MFU=7.0% | step=143236ms | tokens=0.346B | mem=4.2/32.7GB | cost=$169.57/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 02:34 (139.0h) | Phase3: SKIPPED | SFT: Oct 26 07:19 (143.8h) | DPO (Final): Oct 26 07:53 (144.3h) [2026-10-20 07:33:56][INFO][training.pretrain_engine] Starting global step 324 (accumulating 32 micro-batches)... [2026-10-20 07:33:56][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 07:34:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5534 [2026-10-20 07:34:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5006 [2026-10-20 07:34:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5834 [2026-10-20 07:34:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3913 [2026-10-20 07:34:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8618 [2026-10-20 07:34:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1988 [2026-10-20 07:34:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3671 [2026-10-20 07:34:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3418 [2026-10-20 07:34:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0191 [2026-10-20 07:34:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7282 [2026-10-20 07:34:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0589 [2026-10-20 07:34:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.6465 [2026-10-20 07:35:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5004 [2026-10-20 07:35:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2622 [2026-10-20 07:35:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5655 [2026-10-20 07:35:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4716 [2026-10-20 07:35:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5957 [2026-10-20 07:35:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7881 [2026-10-20 07:35:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8684 [2026-10-20 07:35:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1142 [2026-10-20 07:35:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5832 [2026-10-20 07:35:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6573 [2026-10-20 07:35:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7993 [2026-10-20 07:35:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5419 [2026-10-20 07:36:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1256 [2026-10-20 07:36:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3412 [2026-10-20 07:36:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7301 [2026-10-20 07:36:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3370 [2026-10-20 07:36:15][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,514 | MFU: 1.8% [2026-10-20 07:36:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4909 [2026-10-20 07:36:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4552 [2026-10-20 07:36:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7741 [2026-10-20 07:36:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4956 [2026-10-20 07:36:20][INFO][training.pretrain_engine] [Ph2|Step 324] loss=6.5403 | ppl=692.48 | aux=0.02513 | lr=2.17e-04 | gnorm=1.469 | tok/s=29,280 | MFU=7.0% | step=143250ms | tokens=0.347B | mem=4.2/32.7GB | cost=$170.09/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 02:37 (139.0h) | Phase3: SKIPPED | SFT: Oct 26 07:21 (143.8h) | DPO (Final): Oct 26 07:55 (144.3h) [2026-10-20 07:36:23][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step324 | Tokens: 0.347B | Spent: $170.11 | Remaining: $229.89 [2026-10-20 07:36:23][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step324 [2026-10-20 07:36:23][INFO][training.pretrain_engine] Starting global step 325 (accumulating 32 micro-batches)... [2026-10-20 07:36:23][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 07:36:23][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 07:36:23][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 07:36:23][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 07:36:23][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step324 to Phase-Technologies/forge-500m-pretrain (path: phase2_step324)... [2026-10-20 07:36:23][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 07:36:33][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 07:36:33][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step324 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 07:36:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9521 [2026-10-20 07:36:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5918 [2026-10-20 07:36:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6103 [2026-10-20 07:36:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8122 [2026-10-20 07:36:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5886 [2026-10-20 07:36:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8755 [2026-10-20 07:36:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2734 [2026-10-20 07:36:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3584 [2026-10-20 07:37:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4225 [2026-10-20 07:37:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2260 [2026-10-20 07:37:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2632 [2026-10-20 07:37:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4729 [2026-10-20 07:37:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7446 [2026-10-20 07:37:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1378 [2026-10-20 07:37:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3284 [2026-10-20 07:37:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3760 [2026-10-20 07:37:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8661 [2026-10-20 07:37:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2563 [2026-10-20 07:37:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9506 [2026-10-20 07:37:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2911 [2026-10-20 07:38:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3700 [2026-10-20 07:38:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1296 [2026-10-20 07:38:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3093 [2026-10-20 07:38:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4352 [2026-10-20 07:38:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2866 [2026-10-20 07:38:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2074 [2026-10-20 07:38:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3447 [2026-10-20 07:38:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3448 [2026-10-20 07:38:43][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,515 | MFU: 1.8% [2026-10-20 07:38:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1776 [2026-10-20 07:38:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4155 [2026-10-20 07:38:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3120 [2026-10-20 07:38:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3592 [2026-10-20 07:38:47][INFO][training.pretrain_engine] [Ph2|Step 325] loss=6.5412 | ppl=693.15 | aux=0.02512 | lr=2.18e-04 | gnorm=0.824 | tok/s=29,277 | MFU=7.0% | step=143265ms | tokens=0.348B | mem=4.2/32.7GB | cost=$170.63/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 02:39 (139.0h) | Phase3: SKIPPED | SFT: Oct 26 07:24 (143.8h) | DPO (Final): Oct 26 07:58 (144.3h) [2026-10-20 07:38:47][INFO][training.pretrain_engine] Starting global step 326 (accumulating 32 micro-batches)... [2026-10-20 07:38:47][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 07:39:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9651 [2026-10-20 07:39:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6228 [2026-10-20 07:39:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2328 [2026-10-20 07:39:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9372 [2026-10-20 07:39:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7658 [2026-10-20 07:39:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4706 [2026-10-20 07:39:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5399 [2026-10-20 07:39:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0659 [2026-10-20 07:39:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4061 [2026-10-20 07:39:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7941 [2026-10-20 07:39:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1709 [2026-10-20 07:39:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7446 [2026-10-20 07:39:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3969 [2026-10-20 07:39:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4895 [2026-10-20 07:39:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0542 [2026-10-20 07:39:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4483 [2026-10-20 07:40:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4533 [2026-10-20 07:40:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1214 [2026-10-20 07:40:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0942 [2026-10-20 07:40:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9694 [2026-10-20 07:40:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6122 [2026-10-20 07:40:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2507 [2026-10-20 07:40:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4400 [2026-10-20 07:40:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0377 [2026-10-20 07:40:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5011 [2026-10-20 07:40:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3676 [2026-10-20 07:40:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5047 [2026-10-20 07:40:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5595 [2026-10-20 07:41:07][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,500 | MFU: 1.8% [2026-10-20 07:41:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7979 [2026-10-20 07:41:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4007 [2026-10-20 07:41:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6916 [2026-10-20 07:41:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5223 [2026-10-20 07:41:12][INFO][training.pretrain_engine] [Ph2|Step 326] loss=6.4354 | ppl=623.50 | aux=0.02512 | lr=2.19e-04 | gnorm=1.094 | tok/s=29,273 | MFU=7.0% | step=143284ms | tokens=0.349B | mem=4.2/32.7GB | cost=$171.16/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 02:42 (139.0h) | Phase3: SKIPPED | SFT: Oct 26 07:27 (143.8h) | DPO (Final): Oct 26 08:01 (144.3h) [2026-10-20 07:41:12][INFO][training.pretrain_engine] Starting global step 327 (accumulating 32 micro-batches)... [2026-10-20 07:41:12][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 07:41:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1799 [2026-10-20 07:41:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5817 [2026-10-20 07:41:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4796 [2026-10-20 07:41:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5381 [2026-10-20 07:41:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4194 [2026-10-20 07:41:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6321 [2026-10-20 07:41:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0564 [2026-10-20 07:41:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9749 [2026-10-20 07:42:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4436 [2026-10-20 07:42:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3324 [2026-10-20 07:42:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0004 [2026-10-20 07:42:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4268 [2026-10-20 07:42:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1736 [2026-10-20 07:42:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6435 [2026-10-20 07:42:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1591 [2026-10-20 07:42:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9750 [2026-10-20 07:42:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5785 [2026-10-20 07:42:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2096 [2026-10-20 07:42:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0124 [2026-10-20 07:42:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8259 [2026-10-20 07:42:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9429 [2026-10-20 07:42:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5809 [2026-10-20 07:42:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7192 [2026-10-20 07:42:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7297 [2026-10-20 07:43:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3047 [2026-10-20 07:43:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8832 [2026-10-20 07:43:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1689 [2026-10-20 07:43:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0273 [2026-10-20 07:43:31][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,536 | MFU: 1.8% [2026-10-20 07:43:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5336 [2026-10-20 07:43:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8585 [2026-10-20 07:43:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8181 [2026-10-20 07:43:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7048 [2026-10-20 07:43:35][INFO][training.pretrain_engine] [Ph2|Step 327] loss=6.4723 | ppl=646.95 | aux=0.02515 | lr=2.19e-04 | gnorm=1.969 | tok/s=29,272 | MFU=7.0% | step=143286ms | tokens=0.350B | mem=4.2/32.7GB | cost=$171.69/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 02:44 (139.0h) | Phase3: SKIPPED | SFT: Oct 26 07:29 (143.8h) | DPO (Final): Oct 26 08:03 (144.3h) [2026-10-20 07:43:35][INFO][training.pretrain_engine] Starting global step 328 (accumulating 32 micro-batches)... [2026-10-20 07:43:35][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 07:43:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3952 [2026-10-20 07:43:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1218 [2026-10-20 07:43:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9318 [2026-10-20 07:43:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4600 [2026-10-20 07:44:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2021 [2026-10-20 07:44:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9629 [2026-10-20 07:44:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1706 [2026-10-20 07:44:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4245 [2026-10-20 07:44:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5752 [2026-10-20 07:44:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4900 [2026-10-20 07:44:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4687 [2026-10-20 07:44:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6584 [2026-10-20 07:44:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2564 [2026-10-20 07:44:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1069 [2026-10-20 07:44:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4052 [2026-10-20 07:44:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2793 [2026-10-20 07:45:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0915 [2026-10-20 07:45:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7178 [2026-10-20 07:45:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1094 [2026-10-20 07:45:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.4768 [2026-10-20 07:45:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4837 [2026-10-20 07:45:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6847 [2026-10-20 07:45:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4381 [2026-10-20 07:45:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7088 [2026-10-20 07:45:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4887 [2026-10-20 07:45:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3854 [2026-10-20 07:45:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4363 [2026-10-20 07:45:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0756 [2026-10-20 07:45:54][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,566 | MFU: 1.8% [2026-10-20 07:45:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4055 [2026-10-20 07:45:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5270 [2026-10-20 07:45:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3018 [2026-10-20 07:45:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6248 [2026-10-20 07:45:59][INFO][training.pretrain_engine] [Ph2|Step 328] loss=6.5439 | ppl=694.98 | aux=0.02513 | lr=2.20e-04 | gnorm=0.906 | tok/s=29,271 | MFU=7.0% | step=143291ms | tokens=0.351B | mem=4.2/32.7GB | cost=$172.21/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 02:46 (139.0h) | Phase3: SKIPPED | SFT: Oct 26 07:31 (143.8h) | DPO (Final): Oct 26 08:05 (144.3h) [2026-10-20 07:45:59][INFO][training.pretrain_engine] Starting global step 329 (accumulating 32 micro-batches)... [2026-10-20 07:45:59][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 07:46:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8485 [2026-10-20 07:46:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3461 [2026-10-20 07:46:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7673 [2026-10-20 07:46:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2453 [2026-10-20 07:46:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0387 [2026-10-20 07:46:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6048 [2026-10-20 07:46:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2135 [2026-10-20 07:46:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6191 [2026-10-20 07:46:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6288 [2026-10-20 07:46:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8252 [2026-10-20 07:46:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3829 [2026-10-20 07:46:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2129 [2026-10-20 07:47:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5908 [2026-10-20 07:47:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5603 [2026-10-20 07:47:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1351 [2026-10-20 07:47:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7251 [2026-10-20 07:47:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0966 [2026-10-20 07:47:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4429 [2026-10-20 07:47:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6337 [2026-10-20 07:47:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7118 [2026-10-20 07:47:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4545 [2026-10-20 07:47:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2336 [2026-10-20 07:47:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2918 [2026-10-20 07:47:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4760 [2026-10-20 07:48:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5846 [2026-10-20 07:48:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3725 [2026-10-20 07:48:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1417 [2026-10-20 07:48:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4156 [2026-10-20 07:48:17][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,593 | MFU: 1.8% [2026-10-20 07:48:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5553 [2026-10-20 07:48:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0592 [2026-10-20 07:48:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0556 [2026-10-20 07:48:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3233 [2026-10-20 07:48:22][INFO][training.pretrain_engine] [Ph2|Step 329] loss=6.5107 | ppl=672.28 | aux=0.02512 | lr=2.20e-04 | gnorm=1.656 | tok/s=29,270 | MFU=7.0% | step=143295ms | tokens=0.352B | mem=4.2/32.7GB | cost=$172.73/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 02:48 (139.0h) | Phase3: SKIPPED | SFT: Oct 26 07:33 (143.8h) | DPO (Final): Oct 26 08:07 (144.3h) [2026-10-20 07:48:22][INFO][training.pretrain_engine] Starting global step 330 (accumulating 32 micro-batches)... [2026-10-20 07:48:22][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 07:48:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4941 [2026-10-20 07:48:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5363 [2026-10-20 07:48:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2394 [2026-10-20 07:48:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2648 [2026-10-20 07:48:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2765 [2026-10-20 07:48:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3989 [2026-10-20 07:48:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5812 [2026-10-20 07:48:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4750 [2026-10-20 07:49:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8424 [2026-10-20 07:49:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8695 [2026-10-20 07:49:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5947 [2026-10-20 07:49:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9193 [2026-10-20 07:49:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3578 [2026-10-20 07:49:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5038 [2026-10-20 07:49:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1589 [2026-10-20 07:49:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6140 [2026-10-20 07:49:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4189 [2026-10-20 07:49:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4525 [2026-10-20 07:49:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2812 [2026-10-20 07:49:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9262 [2026-10-20 07:50:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5576 [2026-10-20 07:50:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1513 [2026-10-20 07:50:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3047 [2026-10-20 07:50:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2620 [2026-10-20 07:50:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3466 [2026-10-20 07:50:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9466 [2026-10-20 07:50:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0264 [2026-10-20 07:50:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7800 [2026-10-20 07:50:41][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,580 | MFU: 1.8% [2026-10-20 07:50:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8129 [2026-10-20 07:50:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.1245 [2026-10-20 07:50:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5925 [2026-10-20 07:50:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0632 [2026-10-20 07:50:46][INFO][training.pretrain_engine] [Ph2|Step 330] loss=6.4668 | ppl=643.43 | aux=0.02513 | lr=2.21e-04 | gnorm=1.062 | tok/s=29,269 | MFU=7.0% | step=143303ms | tokens=0.353B | mem=4.2/32.7GB | cost=$173.26/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 02:51 (139.0h) | Phase3: SKIPPED | SFT: Oct 26 07:35 (143.8h) | DPO (Final): Oct 26 08:09 (144.3h) [2026-10-20 07:50:46][INFO][training.pretrain_engine] Starting global step 331 (accumulating 32 micro-batches)... [2026-10-20 07:50:46][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 07:51:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1239 [2026-10-20 07:51:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.3241 [2026-10-20 07:51:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5300 [2026-10-20 07:51:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0250 [2026-10-20 07:51:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1876 [2026-10-20 07:51:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9066 [2026-10-20 07:51:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4185 [2026-10-20 07:51:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5205 [2026-10-20 07:51:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4534 [2026-10-20 07:51:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3399 [2026-10-20 07:51:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5801 [2026-10-20 07:51:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3515 [2026-10-20 07:51:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5403 [2026-10-20 07:51:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4537 [2026-10-20 07:51:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3219 [2026-10-20 07:51:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1280 [2026-10-20 07:52:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3303 [2026-10-20 07:52:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6877 [2026-10-20 07:52:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5627 [2026-10-20 07:52:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4850 [2026-10-20 07:52:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5630 [2026-10-20 07:52:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5018 [2026-10-20 07:52:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5113 [2026-10-20 07:52:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1096 [2026-10-20 07:52:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0402 [2026-10-20 07:52:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5514 [2026-10-20 07:52:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5290 [2026-10-20 07:52:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4224 [2026-10-20 07:53:04][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,585 | MFU: 1.8% [2026-10-20 07:53:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5448 [2026-10-20 07:53:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2849 [2026-10-20 07:53:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9690 [2026-10-20 07:53:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9601 [2026-10-20 07:53:10][INFO][training.pretrain_engine] [Ph2|Step 331] loss=6.4938 | ppl=661.02 | aux=0.02514 | lr=2.21e-04 | gnorm=2.188 | tok/s=29,266 | MFU=7.0% | step=143314ms | tokens=0.354B | mem=4.2/32.7GB | cost=$173.79/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 02:53 (139.0h) | Phase3: SKIPPED | SFT: Oct 26 07:38 (143.8h) | DPO (Final): Oct 26 08:12 (144.3h) [2026-10-20 07:53:10][INFO][training.pretrain_engine] Starting global step 332 (accumulating 32 micro-batches)... [2026-10-20 07:53:10][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 07:53:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9760 [2026-10-20 07:53:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6302 [2026-10-20 07:53:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3476 [2026-10-20 07:53:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0638 [2026-10-20 07:53:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4080 [2026-10-20 07:53:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4503 [2026-10-20 07:53:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3302 [2026-10-20 07:53:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3259 [2026-10-20 07:54:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1122 [2026-10-20 07:54:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9719 [2026-10-20 07:54:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3012 [2026-10-20 07:54:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.3082 [2026-10-20 07:54:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3346 [2026-10-20 07:54:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1275 [2026-10-20 07:54:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0339 [2026-10-20 07:54:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1763 [2026-10-20 07:54:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9517 [2026-10-20 07:54:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3247 [2026-10-20 07:54:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0824 [2026-10-20 07:54:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3663 [2026-10-20 07:54:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3136 [2026-10-20 07:54:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1828 [2026-10-20 07:54:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7619 [2026-10-20 07:54:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2723 [2026-10-20 07:55:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5143 [2026-10-20 07:55:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7710 [2026-10-20 07:55:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6556 [2026-10-20 07:55:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9149 [2026-10-20 07:55:29][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,563 | MFU: 1.8% [2026-10-20 07:55:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6566 [2026-10-20 07:55:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5885 [2026-10-20 07:55:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2729 [2026-10-20 07:55:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8698 [2026-10-20 07:55:34][INFO][training.pretrain_engine] [Ph2|Step 332] loss=6.5234 | ppl=680.88 | aux=0.02514 | lr=2.22e-04 | gnorm=1.727 | tok/s=29,265 | MFU=7.0% | step=143321ms | tokens=0.355B | mem=4.2/32.7GB | cost=$174.31/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 02:55 (139.0h) | Phase3: SKIPPED | SFT: Oct 26 07:40 (143.7h) | DPO (Final): Oct 26 08:14 (144.3h) [2026-10-20 07:55:34][INFO][training.pretrain_engine] Starting global step 333 (accumulating 32 micro-batches)... [2026-10-20 07:55:34][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 07:55:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6251 [2026-10-20 07:55:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6218 [2026-10-20 07:55:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7166 [2026-10-20 07:55:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5364 [2026-10-20 07:56:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8596 [2026-10-20 07:56:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9651 [2026-10-20 07:56:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9327 [2026-10-20 07:56:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7067 [2026-10-20 07:56:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3806 [2026-10-20 07:56:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2452 [2026-10-20 07:56:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3596 [2026-10-20 07:56:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0799 [2026-10-20 07:56:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4602 [2026-10-20 07:56:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3921 [2026-10-20 07:56:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0834 [2026-10-20 07:56:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6949 [2026-10-20 07:57:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6687 [2026-10-20 07:57:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1769 [2026-10-20 07:57:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5785 [2026-10-20 07:57:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6839 [2026-10-20 07:57:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2771 [2026-10-20 07:57:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8729 [2026-10-20 07:57:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2183 [2026-10-20 07:57:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9633 [2026-10-20 07:57:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7249 [2026-10-20 07:57:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4537 [2026-10-20 07:57:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4333 [2026-10-20 07:57:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4568 [2026-10-20 07:57:53][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,523 | MFU: 1.8% [2026-10-20 07:57:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.1390 [2026-10-20 07:57:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8377 [2026-10-20 07:57:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7790 [2026-10-20 07:57:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0405 [2026-10-20 07:57:58][INFO][training.pretrain_engine] [Ph2|Step 333] loss=6.4503 | ppl=632.88 | aux=0.02512 | lr=2.23e-04 | gnorm=1.828 | tok/s=29,263 | MFU=7.0% | step=143332ms | tokens=0.357B | mem=4.2/32.7GB | cost=$174.84/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 02:58 (139.0h) | Phase3: SKIPPED | SFT: Oct 26 07:42 (143.7h) | DPO (Final): Oct 26 08:17 (144.3h) [2026-10-20 07:58:01][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step333 | Tokens: 0.357B | Spent: $174.85 | Remaining: $225.15 [2026-10-20 07:58:01][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step333 [2026-10-20 07:58:01][INFO][training.pretrain_engine] Starting global step 334 (accumulating 32 micro-batches)... [2026-10-20 07:58:01][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 07:58:01][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 07:58:01][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 07:58:01][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 07:58:01][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step333 to Phase-Technologies/forge-500m-pretrain (path: phase2_step333)... [2026-10-20 07:58:01][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 07:58:11][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 07:58:11][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step333 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 07:58:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6618 [2026-10-20 07:58:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6670 [2026-10-20 07:58:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2501 [2026-10-20 07:58:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3676 [2026-10-20 07:58:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4654 [2026-10-20 07:58:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9191 [2026-10-20 07:58:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3887 [2026-10-20 07:58:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5713 [2026-10-20 07:58:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1059 [2026-10-20 07:58:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7681 [2026-10-20 07:58:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5804 [2026-10-20 07:58:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3814 [2026-10-20 07:59:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5585 [2026-10-20 07:59:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3299 [2026-10-20 07:59:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3744 [2026-10-20 07:59:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2276 [2026-10-20 07:59:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5661 [2026-10-20 07:59:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5654 [2026-10-20 07:59:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3980 [2026-10-20 07:59:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6310 [2026-10-20 07:59:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2971 [2026-10-20 07:59:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5084 [2026-10-20 07:59:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2673 [2026-10-20 07:59:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7602 [2026-10-20 08:00:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6586 [2026-10-20 08:00:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7781 [2026-10-20 08:00:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7089 [2026-10-20 08:00:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7124 [2026-10-20 08:00:20][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,529 | MFU: 1.8% [2026-10-20 08:00:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5390 [2026-10-20 08:00:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1904 [2026-10-20 08:00:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4314 [2026-10-20 08:00:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4938 [2026-10-20 08:00:25][INFO][training.pretrain_engine] [Ph2|Step 334] loss=6.4343 | ppl=622.85 | aux=0.02512 | lr=2.23e-04 | gnorm=1.656 | tok/s=29,261 | MFU=7.0% | step=143339ms | tokens=0.358B | mem=4.2/32.7GB | cost=$175.38/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 03:00 (139.0h) | Phase3: SKIPPED | SFT: Oct 26 07:45 (143.7h) | DPO (Final): Oct 26 08:19 (144.3h) [2026-10-20 08:00:25][INFO][training.pretrain_engine] Starting global step 335 (accumulating 32 micro-batches)... [2026-10-20 08:00:25][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 08:00:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5805 [2026-10-20 08:00:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1711 [2026-10-20 08:00:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3790 [2026-10-20 08:00:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1455 [2026-10-20 08:01:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1449 [2026-10-20 08:01:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4570 [2026-10-20 08:01:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5307 [2026-10-20 08:01:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3288 [2026-10-20 08:01:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9008 [2026-10-20 08:01:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0696 [2026-10-20 08:01:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3353 [2026-10-20 08:01:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7543 [2026-10-20 08:01:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7587 [2026-10-20 08:01:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2643 [2026-10-20 08:01:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4128 [2026-10-20 08:01:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6884 [2026-10-20 08:01:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5827 [2026-10-20 08:01:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6644 [2026-10-20 08:01:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5722 [2026-10-20 08:01:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8171 [2026-10-20 08:02:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2744 [2026-10-20 08:02:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1062 [2026-10-20 08:02:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6479 [2026-10-20 08:02:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8752 [2026-10-20 08:02:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7889 [2026-10-20 08:02:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0645 [2026-10-20 08:02:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7654 [2026-10-20 08:02:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7932 [2026-10-20 08:02:44][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,547 | MFU: 1.8% [2026-10-20 08:02:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3555 [2026-10-20 08:02:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5276 [2026-10-20 08:02:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4966 [2026-10-20 08:02:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7908 [2026-10-20 08:02:48][INFO][training.pretrain_engine] [Ph2|Step 335] loss=6.4578 | ppl=637.63 | aux=0.02513 | lr=2.24e-04 | gnorm=1.320 | tok/s=29,260 | MFU=7.0% | step=143348ms | tokens=0.359B | mem=4.2/32.7GB | cost=$175.90/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 03:02 (139.0h) | Phase3: SKIPPED | SFT: Oct 26 07:47 (143.7h) | DPO (Final): Oct 26 08:21 (144.3h) [2026-10-20 08:02:48][INFO][training.pretrain_engine] Starting global step 336 (accumulating 32 micro-batches)... [2026-10-20 08:02:48][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 08:03:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5488 [2026-10-20 08:03:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4633 [2026-10-20 08:03:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3977 [2026-10-20 08:03:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5040 [2026-10-20 08:03:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0245 [2026-10-20 08:03:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7110 [2026-10-20 08:03:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2388 [2026-10-20 08:03:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8151 [2026-10-20 08:03:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4277 [2026-10-20 08:03:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5442 [2026-10-20 08:03:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3634 [2026-10-20 08:03:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7733 [2026-10-20 08:04:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6333 [2026-10-20 08:04:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3709 [2026-10-20 08:04:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4132 [2026-10-20 08:04:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2930 [2026-10-20 08:04:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4130 [2026-10-20 08:04:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2364 [2026-10-20 08:04:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8708 [2026-10-20 08:04:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2849 [2026-10-20 08:04:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8642 [2026-10-20 08:04:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2999 [2026-10-20 08:04:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7467 [2026-10-20 08:04:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5022 [2026-10-20 08:04:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7031 [2026-10-20 08:04:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8535 [2026-10-20 08:04:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7080 [2026-10-20 08:04:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5590 [2026-10-20 08:05:07][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,549 | MFU: 1.8% [2026-10-20 08:05:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5279 [2026-10-20 08:05:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5177 [2026-10-20 08:05:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4019 [2026-10-20 08:05:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5180 [2026-10-20 08:05:13][INFO][training.pretrain_engine] [Ph2|Step 336] loss=6.4986 | ppl=664.21 | aux=0.02513 | lr=2.24e-04 | gnorm=2.062 | tok/s=29,258 | MFU=7.0% | step=143357ms | tokens=0.360B | mem=4.2/32.7GB | cost=$176.43/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 03:05 (139.0h) | Phase3: SKIPPED | SFT: Oct 26 07:49 (143.7h) | DPO (Final): Oct 26 08:24 (144.3h) [2026-10-20 08:05:13][INFO][training.pretrain_engine] Starting global step 337 (accumulating 32 micro-batches)... [2026-10-20 08:05:13][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 08:05:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3315 [2026-10-20 08:05:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4832 [2026-10-20 08:05:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1382 [2026-10-20 08:05:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2710 [2026-10-20 08:05:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4701 [2026-10-20 08:05:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9995 [2026-10-20 08:05:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6231 [2026-10-20 08:05:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7396 [2026-10-20 08:06:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2197 [2026-10-20 08:06:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0043 [2026-10-20 08:06:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5597 [2026-10-20 08:06:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9953 [2026-10-20 08:06:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1684 [2026-10-20 08:06:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1787 [2026-10-20 08:06:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6003 [2026-10-20 08:06:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7775 [2026-10-20 08:06:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8152 [2026-10-20 08:06:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2439 [2026-10-20 08:06:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8650 [2026-10-20 08:06:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7088 [2026-10-20 08:07:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8933 [2026-10-20 08:07:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7585 [2026-10-20 08:07:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5731 [2026-10-20 08:07:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5942 [2026-10-20 08:07:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4153 [2026-10-20 08:07:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4100 [2026-10-20 08:07:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5236 [2026-10-20 08:07:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7250 [2026-10-20 08:07:31][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,562 | MFU: 1.8% [2026-10-20 08:07:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4435 [2026-10-20 08:07:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6976 [2026-10-20 08:07:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4229 [2026-10-20 08:07:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0641 [2026-10-20 08:07:36][INFO][training.pretrain_engine] [Ph2|Step 337] loss=6.5746 | ppl=716.68 | aux=0.02515 | lr=2.25e-04 | gnorm=1.414 | tok/s=29,258 | MFU=7.0% | step=143355ms | tokens=0.361B | mem=4.2/32.7GB | cost=$176.95/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 03:06 (139.0h) | Phase3: SKIPPED | SFT: Oct 26 07:51 (143.7h) | DPO (Final): Oct 26 08:25 (144.3h) [2026-10-20 08:07:36][INFO][training.pretrain_engine] Starting global step 338 (accumulating 32 micro-batches)... [2026-10-20 08:07:36][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 08:07:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5306 [2026-10-20 08:07:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9537 [2026-10-20 08:07:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9895 [2026-10-20 08:07:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2761 [2026-10-20 08:08:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3969 [2026-10-20 08:08:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1111 [2026-10-20 08:08:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2172 [2026-10-20 08:08:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0140 [2026-10-20 08:08:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9106 [2026-10-20 08:08:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5733 [2026-10-20 08:08:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0234 [2026-10-20 08:08:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6692 [2026-10-20 08:08:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0690 [2026-10-20 08:08:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2162 [2026-10-20 08:08:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2396 [2026-10-20 08:08:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6908 [2026-10-20 08:09:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4725 [2026-10-20 08:09:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2661 [2026-10-20 08:09:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8849 [2026-10-20 08:09:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3906 [2026-10-20 08:09:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2146 [2026-10-20 08:09:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3795 [2026-10-20 08:09:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2368 [2026-10-20 08:09:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8921 [2026-10-20 08:09:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8262 [2026-10-20 08:09:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5142 [2026-10-20 08:09:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7104 [2026-10-20 08:09:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5008 [2026-10-20 08:09:55][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,565 | MFU: 1.8% [2026-10-20 08:09:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0272 [2026-10-20 08:09:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3454 [2026-10-20 08:09:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2277 [2026-10-20 08:09:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.4824 [2026-10-20 08:09:59][INFO][training.pretrain_engine] [Ph2|Step 338] loss=6.6601 | ppl=780.60 | aux=0.02513 | lr=2.25e-04 | gnorm=1.352 | tok/s=29,259 | MFU=7.0% | step=143351ms | tokens=0.362B | mem=4.2/32.7GB | cost=$177.48/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 03:08 (139.0h) | Phase3: SKIPPED | SFT: Oct 26 07:53 (143.7h) | DPO (Final): Oct 26 08:27 (144.3h) [2026-10-20 08:09:59][INFO][training.pretrain_engine] Starting global step 339 (accumulating 32 micro-batches)... [2026-10-20 08:09:59][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 08:10:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2516 [2026-10-20 08:10:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9031 [2026-10-20 08:10:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3320 [2026-10-20 08:10:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2305 [2026-10-20 08:10:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3989 [2026-10-20 08:10:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7385 [2026-10-20 08:10:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2768 [2026-10-20 08:10:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6723 [2026-10-20 08:10:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3830 [2026-10-20 08:10:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0287 [2026-10-20 08:10:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3131 [2026-10-20 08:10:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5430 [2026-10-20 08:11:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2385 [2026-10-20 08:11:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1809 [2026-10-20 08:11:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8682 [2026-10-20 08:11:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3251 [2026-10-20 08:11:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2865 [2026-10-20 08:11:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6063 [2026-10-20 08:11:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3358 [2026-10-20 08:11:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7254 [2026-10-20 08:11:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.3208 [2026-10-20 08:11:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0614 [2026-10-20 08:11:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2502 [2026-10-20 08:11:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2382 [2026-10-20 08:12:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9243 [2026-10-20 08:12:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1088 [2026-10-20 08:12:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8300 [2026-10-20 08:12:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9833 [2026-10-20 08:12:18][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,579 | MFU: 1.8% [2026-10-20 08:12:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1291 [2026-10-20 08:12:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3511 [2026-10-20 08:12:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9787 [2026-10-20 08:12:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1130 [2026-10-20 08:12:23][INFO][training.pretrain_engine] [Ph2|Step 339] loss=6.6153 | ppl=746.45 | aux=0.02516 | lr=2.26e-04 | gnorm=1.570 | tok/s=29,258 | MFU=7.0% | step=143357ms | tokens=0.363B | mem=4.2/32.7GB | cost=$178.00/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 03:10 (139.0h) | Phase3: SKIPPED | SFT: Oct 26 07:55 (143.7h) | DPO (Final): Oct 26 08:29 (144.3h) [2026-10-20 08:12:23][INFO][training.pretrain_engine] Starting global step 340 (accumulating 32 micro-batches)... [2026-10-20 08:12:23][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 08:12:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2731 [2026-10-20 08:12:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.4223 [2026-10-20 08:12:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4964 [2026-10-20 08:12:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4310 [2026-10-20 08:12:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7414 [2026-10-20 08:12:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4762 [2026-10-20 08:12:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6150 [2026-10-20 08:12:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4461 [2026-10-20 08:13:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5324 [2026-10-20 08:13:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7479 [2026-10-20 08:13:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6617 [2026-10-20 08:13:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0851 [2026-10-20 08:13:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9915 [2026-10-20 08:13:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7435 [2026-10-20 08:13:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3970 [2026-10-20 08:13:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3379 [2026-10-20 08:13:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3373 [2026-10-20 08:13:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3868 [2026-10-20 08:13:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6632 [2026-10-20 08:13:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4192 [2026-10-20 08:14:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2201 [2026-10-20 08:14:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8116 [2026-10-20 08:14:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6705 [2026-10-20 08:14:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6834 [2026-10-20 08:14:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8964 [2026-10-20 08:14:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9835 [2026-10-20 08:14:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6761 [2026-10-20 08:14:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9047 [2026-10-20 08:14:42][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,575 | MFU: 1.8% [2026-10-20 08:14:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9121 [2026-10-20 08:14:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0801 [2026-10-20 08:14:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3588 [2026-10-20 08:14:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7103 [2026-10-20 08:14:47][INFO][training.pretrain_engine] [Ph2|Step 340] loss=6.4118 | ppl=609.00 | aux=0.02517 | lr=2.26e-04 | gnorm=1.188 | tok/s=29,257 | MFU=7.0% | step=143362ms | tokens=0.364B | mem=4.2/32.7GB | cost=$178.53/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 03:12 (139.0h) | Phase3: SKIPPED | SFT: Oct 26 07:57 (143.7h) | DPO (Final): Oct 26 08:31 (144.3h) [2026-10-20 08:14:47][INFO][training.pretrain_engine] Starting global step 341 (accumulating 32 micro-batches)... [2026-10-20 08:14:47][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 08:15:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8603 [2026-10-20 08:15:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5596 [2026-10-20 08:15:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1316 [2026-10-20 08:15:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7604 [2026-10-20 08:15:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1201 [2026-10-20 08:15:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5584 [2026-10-20 08:15:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2796 [2026-10-20 08:15:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4540 [2026-10-20 08:15:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5118 [2026-10-20 08:15:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4573 [2026-10-20 08:15:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6759 [2026-10-20 08:15:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1757 [2026-10-20 08:15:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9443 [2026-10-20 08:15:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1465 [2026-10-20 08:15:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8241 [2026-10-20 08:15:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8745 [2026-10-20 08:16:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4022 [2026-10-20 08:16:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2793 [2026-10-20 08:16:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0809 [2026-10-20 08:16:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1297 [2026-10-20 08:16:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2708 [2026-10-20 08:16:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5758 [2026-10-20 08:16:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5251 [2026-10-20 08:16:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3685 [2026-10-20 08:16:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6426 [2026-10-20 08:16:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0292 [2026-10-20 08:16:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2000 [2026-10-20 08:16:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6579 [2026-10-20 08:17:05][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,572 | MFU: 1.8% [2026-10-20 08:17:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9786 [2026-10-20 08:17:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0594 [2026-10-20 08:17:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1789 [2026-10-20 08:17:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6777 [2026-10-20 08:17:11][INFO][training.pretrain_engine] [Ph2|Step 341] loss=6.4519 | ppl=633.91 | aux=0.02511 | lr=2.27e-04 | gnorm=1.172 | tok/s=29,255 | MFU=7.0% | step=143370ms | tokens=0.365B | mem=4.2/32.7GB | cost=$179.05/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 03:14 (139.0h) | Phase3: SKIPPED | SFT: Oct 26 07:59 (143.7h) | DPO (Final): Oct 26 08:33 (144.3h) [2026-10-20 08:17:11][INFO][training.pretrain_engine] Starting global step 342 (accumulating 32 micro-batches)... [2026-10-20 08:17:11][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 08:17:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0778 [2026-10-20 08:17:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0358 [2026-10-20 08:17:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5899 [2026-10-20 08:17:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2773 [2026-10-20 08:17:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8815 [2026-10-20 08:17:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4712 [2026-10-20 08:17:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2529 [2026-10-20 08:17:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2590 [2026-10-20 08:18:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4561 [2026-10-20 08:18:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5172 [2026-10-20 08:18:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0465 [2026-10-20 08:18:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6051 [2026-10-20 08:18:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3199 [2026-10-20 08:18:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0168 [2026-10-20 08:18:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2425 [2026-10-20 08:18:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9656 [2026-10-20 08:18:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4337 [2026-10-20 08:18:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1425 [2026-10-20 08:18:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4397 [2026-10-20 08:18:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1159 [2026-10-20 08:18:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0105 [2026-10-20 08:18:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7154 [2026-10-20 08:18:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6879 [2026-10-20 08:18:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9029 [2026-10-20 08:19:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7588 [2026-10-20 08:19:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7122 [2026-10-20 08:19:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4600 [2026-10-20 08:19:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7386 [2026-10-20 08:19:30][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,535 | MFU: 1.8% [2026-10-20 08:19:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2940 [2026-10-20 08:19:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7868 [2026-10-20 08:19:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2998 [2026-10-20 08:19:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8403 [2026-10-20 08:19:34][INFO][training.pretrain_engine] [Ph2|Step 342] loss=6.3717 | ppl=585.03 | aux=0.02516 | lr=2.28e-04 | gnorm=2.031 | tok/s=29,254 | MFU=7.0% | step=143377ms | tokens=0.366B | mem=4.2/32.7GB | cost=$179.58/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 03:17 (139.0h) | Phase3: SKIPPED | SFT: Oct 26 08:01 (143.7h) | DPO (Final): Oct 26 08:36 (144.3h) [2026-10-20 08:19:38][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step342 | Tokens: 0.366B | Spent: $179.59 | Remaining: $220.41 [2026-10-20 08:19:38][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step342 [2026-10-20 08:19:38][INFO][training.pretrain_engine] Starting global step 343 (accumulating 32 micro-batches)... [2026-10-20 08:19:38][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 08:19:38][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 08:19:38][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 08:19:38][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 08:19:38][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step342 to Phase-Technologies/forge-500m-pretrain (path: phase2_step342)... [2026-10-20 08:19:38][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 08:19:48][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 08:19:48][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step342 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 08:19:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3470 [2026-10-20 08:19:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2882 [2026-10-20 08:19:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5120 [2026-10-20 08:19:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5964 [2026-10-20 08:20:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6588 [2026-10-20 08:20:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4410 [2026-10-20 08:20:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9916 [2026-10-20 08:20:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1679 [2026-10-20 08:20:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6207 [2026-10-20 08:20:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5545 [2026-10-20 08:20:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3927 [2026-10-20 08:20:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6318 [2026-10-20 08:20:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0360 [2026-10-20 08:20:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4146 [2026-10-20 08:20:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0878 [2026-10-20 08:20:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7732 [2026-10-20 08:21:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6483 [2026-10-20 08:21:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5072 [2026-10-20 08:21:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2581 [2026-10-20 08:21:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3378 [2026-10-20 08:21:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4879 [2026-10-20 08:21:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3797 [2026-10-20 08:21:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3293 [2026-10-20 08:21:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4830 [2026-10-20 08:21:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1140 [2026-10-20 08:21:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2500 [2026-10-20 08:21:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3462 [2026-10-20 08:21:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1621 [2026-10-20 08:21:57][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,518 | MFU: 1.8% [2026-10-20 08:22:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5374 [2026-10-20 08:22:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5606 [2026-10-20 08:22:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8065 [2026-10-20 08:22:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3435 [2026-10-20 08:22:02][INFO][training.pretrain_engine] [Ph2|Step 343] loss=6.3855 | ppl=593.19 | aux=0.02513 | lr=2.28e-04 | gnorm=1.203 | tok/s=29,253 | MFU=7.0% | step=143381ms | tokens=0.367B | mem=4.2/32.7GB | cost=$180.12/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 03:19 (139.0h) | Phase3: SKIPPED | SFT: Oct 26 08:04 (143.7h) | DPO (Final): Oct 26 08:38 (144.3h) [2026-10-20 08:22:02][INFO][training.pretrain_engine] Starting global step 344 (accumulating 32 micro-batches)... [2026-10-20 08:22:02][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 08:22:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3977 [2026-10-20 08:22:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3468 [2026-10-20 08:22:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3885 [2026-10-20 08:22:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6215 [2026-10-20 08:22:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9813 [2026-10-20 08:22:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6896 [2026-10-20 08:22:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3804 [2026-10-20 08:22:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1689 [2026-10-20 08:22:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4460 [2026-10-20 08:22:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2048 [2026-10-20 08:22:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2027 [2026-10-20 08:22:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4270 [2026-10-20 08:23:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9293 [2026-10-20 08:23:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3456 [2026-10-20 08:23:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7566 [2026-10-20 08:23:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0386 [2026-10-20 08:23:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7765 [2026-10-20 08:23:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3820 [2026-10-20 08:23:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2889 [2026-10-20 08:23:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3691 [2026-10-20 08:23:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0744 [2026-10-20 08:23:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6409 [2026-10-20 08:23:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0786 [2026-10-20 08:23:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5409 [2026-10-20 08:24:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4880 [2026-10-20 08:24:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0349 [2026-10-20 08:24:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4366 [2026-10-20 08:24:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2716 [2026-10-20 08:24:21][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,539 | MFU: 1.8% [2026-10-20 08:24:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5196 [2026-10-20 08:24:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9626 [2026-10-20 08:24:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0997 [2026-10-20 08:24:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1061 [2026-10-20 08:24:25][INFO][training.pretrain_engine] [Ph2|Step 344] loss=6.5158 | ppl=675.70 | aux=0.02513 | lr=2.29e-04 | gnorm=2.453 | tok/s=29,253 | MFU=7.0% | step=143380ms | tokens=0.368B | mem=4.2/32.7GB | cost=$180.64/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 03:20 (138.9h) | Phase3: SKIPPED | SFT: Oct 26 08:05 (143.7h) | DPO (Final): Oct 26 08:39 (144.3h) [2026-10-20 08:24:25][INFO][training.pretrain_engine] Starting global step 345 (accumulating 32 micro-batches)... [2026-10-20 08:24:25][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 08:24:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5997 [2026-10-20 08:24:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7976 [2026-10-20 08:24:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4736 [2026-10-20 08:24:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7420 [2026-10-20 08:25:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3237 [2026-10-20 08:25:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3059 [2026-10-20 08:25:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5002 [2026-10-20 08:25:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0628 [2026-10-20 08:25:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2417 [2026-10-20 08:25:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9269 [2026-10-20 08:25:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3607 [2026-10-20 08:25:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3706 [2026-10-20 08:25:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0336 [2026-10-20 08:25:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9024 [2026-10-20 08:25:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8985 [2026-10-20 08:25:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1693 [2026-10-20 08:25:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6858 [2026-10-20 08:25:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0515 [2026-10-20 08:25:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0943 [2026-10-20 08:25:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5899 [2026-10-20 08:26:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4804 [2026-10-20 08:26:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4906 [2026-10-20 08:26:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9600 [2026-10-20 08:26:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7743 [2026-10-20 08:26:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1855 [2026-10-20 08:26:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3244 [2026-10-20 08:26:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4744 [2026-10-20 08:26:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9775 [2026-10-20 08:26:45][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,488 | MFU: 1.8% [2026-10-20 08:26:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1438 [2026-10-20 08:26:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6925 [2026-10-20 08:26:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9388 [2026-10-20 08:26:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6210 [2026-10-20 08:26:50][INFO][training.pretrain_engine] [Ph2|Step 345] loss=6.5299 | ppl=685.30 | aux=0.02514 | lr=2.29e-04 | gnorm=1.641 | tok/s=29,251 | MFU=7.0% | step=143391ms | tokens=0.369B | mem=4.2/32.7GB | cost=$181.17/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 03:23 (138.9h) | Phase3: SKIPPED | SFT: Oct 26 08:08 (143.7h) | DPO (Final): Oct 26 08:42 (144.3h) [2026-10-20 08:26:50][INFO][training.pretrain_engine] Starting global step 346 (accumulating 32 micro-batches)... [2026-10-20 08:26:50][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 08:27:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2952 [2026-10-20 08:27:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2195 [2026-10-20 08:27:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6839 [2026-10-20 08:27:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5317 [2026-10-20 08:27:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2546 [2026-10-20 08:27:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8110 [2026-10-20 08:27:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2622 [2026-10-20 08:27:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5035 [2026-10-20 08:27:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3090 [2026-10-20 08:27:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9890 [2026-10-20 08:27:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2567 [2026-10-20 08:27:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9655 [2026-10-20 08:28:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4377 [2026-10-20 08:28:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2620 [2026-10-20 08:28:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1161 [2026-10-20 08:28:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7276 [2026-10-20 08:28:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9759 [2026-10-20 08:28:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5133 [2026-10-20 08:28:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7414 [2026-10-20 08:28:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1314 [2026-10-20 08:28:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3280 [2026-10-20 08:28:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3980 [2026-10-20 08:28:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5345 [2026-10-20 08:28:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4170 [2026-10-20 08:28:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1056 [2026-10-20 08:28:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2402 [2026-10-20 08:28:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3196 [2026-10-20 08:28:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0517 [2026-10-20 08:29:09][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,526 | MFU: 1.8% [2026-10-20 08:29:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3215 [2026-10-20 08:29:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2545 [2026-10-20 08:29:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4529 [2026-10-20 08:29:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5743 [2026-10-20 08:29:14][INFO][training.pretrain_engine] [Ph2|Step 346] loss=6.5290 | ppl=684.71 | aux=0.02515 | lr=2.30e-04 | gnorm=2.672 | tok/s=29,250 | MFU=7.0% | step=143396ms | tokens=0.370B | mem=4.2/32.7GB | cost=$181.70/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 03:25 (138.9h) | Phase3: SKIPPED | SFT: Oct 26 08:10 (143.7h) | DPO (Final): Oct 26 08:44 (144.3h) [2026-10-20 08:29:14][INFO][training.pretrain_engine] Starting global step 347 (accumulating 32 micro-batches)... [2026-10-20 08:29:14][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 08:29:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6850 [2026-10-20 08:29:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3745 [2026-10-20 08:29:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9490 [2026-10-20 08:29:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1440 [2026-10-20 08:29:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6976 [2026-10-20 08:29:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6923 [2026-10-20 08:29:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5748 [2026-10-20 08:29:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4169 [2026-10-20 08:30:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6741 [2026-10-20 08:30:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5570 [2026-10-20 08:30:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8134 [2026-10-20 08:30:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9231 [2026-10-20 08:30:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5379 [2026-10-20 08:30:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2124 [2026-10-20 08:30:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7805 [2026-10-20 08:30:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0395 [2026-10-20 08:30:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3173 [2026-10-20 08:30:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5038 [2026-10-20 08:30:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1294 [2026-10-20 08:30:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5531 [2026-10-20 08:31:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3677 [2026-10-20 08:31:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1692 [2026-10-20 08:31:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4329 [2026-10-20 08:31:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7317 [2026-10-20 08:31:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7048 [2026-10-20 08:31:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7764 [2026-10-20 08:31:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9532 [2026-10-20 08:31:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3373 [2026-10-20 08:31:33][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,541 | MFU: 1.8% [2026-10-20 08:31:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2286 [2026-10-20 08:31:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4352 [2026-10-20 08:31:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2011 [2026-10-20 08:31:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.1882 [2026-10-20 08:31:37][INFO][training.pretrain_engine] [Ph2|Step 347] loss=6.4609 | ppl=639.65 | aux=0.02515 | lr=2.30e-04 | gnorm=2.078 | tok/s=29,250 | MFU=7.0% | step=143395ms | tokens=0.371B | mem=4.2/32.7GB | cost=$182.22/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 03:27 (138.9h) | Phase3: SKIPPED | SFT: Oct 26 08:12 (143.7h) | DPO (Final): Oct 26 08:46 (144.2h) [2026-10-20 08:31:37][INFO][training.pretrain_engine] Starting global step 348 (accumulating 32 micro-batches)... [2026-10-20 08:31:37][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 08:31:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4493 [2026-10-20 08:31:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7345 [2026-10-20 08:31:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5424 [2026-10-20 08:31:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2525 [2026-10-20 08:32:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6949 [2026-10-20 08:32:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4874 [2026-10-20 08:32:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4732 [2026-10-20 08:32:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5298 [2026-10-20 08:32:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1942 [2026-10-20 08:32:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1911 [2026-10-20 08:32:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0167 [2026-10-20 08:32:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7902 [2026-10-20 08:32:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1980 [2026-10-20 08:32:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.5766 [2026-10-20 08:32:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.5949 [2026-10-20 08:32:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0372 [2026-10-20 08:33:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.4299 [2026-10-20 08:33:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6269 [2026-10-20 08:33:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7731 [2026-10-20 08:33:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.8029 [2026-10-20 08:33:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3066 [2026-10-20 08:33:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6094 [2026-10-20 08:33:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5530 [2026-10-20 08:33:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3321 [2026-10-20 08:33:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8667 [2026-10-20 08:33:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8367 [2026-10-20 08:33:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9927 [2026-10-20 08:33:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8770 [2026-10-20 08:33:57][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,533 | MFU: 1.8% [2026-10-20 08:34:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4785 [2026-10-20 08:34:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5762 [2026-10-20 08:34:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1914 [2026-10-20 08:34:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4327 [2026-10-20 08:34:01][INFO][training.pretrain_engine] [Ph2|Step 348] loss=6.5168 | ppl=676.42 | aux=0.02515 | lr=2.31e-04 | gnorm=1.836 | tok/s=29,249 | MFU=7.0% | step=143401ms | tokens=0.372B | mem=4.2/32.7GB | cost=$182.75/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 03:29 (138.9h) | Phase3: SKIPPED | SFT: Oct 26 08:14 (143.7h) | DPO (Final): Oct 26 08:48 (144.2h) [2026-10-20 08:34:01][INFO][training.pretrain_engine] Starting global step 349 (accumulating 32 micro-batches)... [2026-10-20 08:34:01][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 08:34:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2457 [2026-10-20 08:34:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2179 [2026-10-20 08:34:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7608 [2026-10-20 08:34:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.2240 [2026-10-20 08:34:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3167 [2026-10-20 08:34:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4166 [2026-10-20 08:34:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3571 [2026-10-20 08:34:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8757 [2026-10-20 08:34:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1685 [2026-10-20 08:34:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5328 [2026-10-20 08:34:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2873 [2026-10-20 08:34:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2873 [2026-10-20 08:35:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5202 [2026-10-20 08:35:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5619 [2026-10-20 08:35:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8440 [2026-10-20 08:35:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4226 [2026-10-20 08:35:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6873 [2026-10-20 08:35:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6183 [2026-10-20 08:35:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6307 [2026-10-20 08:35:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6007 [2026-10-20 08:35:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2717 [2026-10-20 08:35:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3639 [2026-10-20 08:35:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3559 [2026-10-20 08:35:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3068 [2026-10-20 08:36:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2174 [2026-10-20 08:36:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2389 [2026-10-20 08:36:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2870 [2026-10-20 08:36:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1562 [2026-10-20 08:36:20][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,537 | MFU: 1.8% [2026-10-20 08:36:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5087 [2026-10-20 08:36:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4340 [2026-10-20 08:36:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7801 [2026-10-20 08:36:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3984 [2026-10-20 08:36:25][INFO][training.pretrain_engine] [Ph2|Step 349] loss=6.4409 | ppl=626.98 | aux=0.02513 | lr=2.32e-04 | gnorm=1.461 | tok/s=29,249 | MFU=7.0% | step=143402ms | tokens=0.373B | mem=4.2/32.7GB | cost=$183.27/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 03:31 (138.9h) | Phase3: SKIPPED | SFT: Oct 26 08:16 (143.7h) | DPO (Final): Oct 26 08:50 (144.2h) [2026-10-20 08:36:25][INFO][training.pretrain_engine] Starting global step 350 (accumulating 32 micro-batches)... [2026-10-20 08:36:25][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 08:36:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8638 [2026-10-20 08:36:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6362 [2026-10-20 08:36:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9711 [2026-10-20 08:36:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0060 [2026-10-20 08:37:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0574 [2026-10-20 08:37:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2234 [2026-10-20 08:37:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.3844 [2026-10-20 08:37:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1158 [2026-10-20 08:37:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8572 [2026-10-20 08:37:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0287 [2026-10-20 08:37:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.2268 [2026-10-20 08:37:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9078 [2026-10-20 08:37:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6324 [2026-10-20 08:37:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9239 [2026-10-20 08:37:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4638 [2026-10-20 08:37:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2822 [2026-10-20 08:37:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0889 [2026-10-20 08:37:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6191 [2026-10-20 08:37:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8978 [2026-10-20 08:37:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7567 [2026-10-20 08:38:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4236 [2026-10-20 08:38:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0724 [2026-10-20 08:38:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3130 [2026-10-20 08:38:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5137 [2026-10-20 08:38:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4962 [2026-10-20 08:38:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8859 [2026-10-20 08:38:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4808 [2026-10-20 08:38:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9483 [2026-10-20 08:38:44][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,514 | MFU: 1.8% [2026-10-20 08:38:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.9351 [2026-10-20 08:38:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4313 [2026-10-20 08:38:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2349 [2026-10-20 08:38:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5101 [2026-10-20 08:38:49][INFO][training.pretrain_engine] [Ph2|Step 350] loss=6.5191 | ppl=677.94 | aux=0.02514 | lr=2.32e-04 | gnorm=1.797 | tok/s=29,246 | MFU=7.0% | step=143414ms | tokens=0.374B | mem=4.2/32.7GB | cost=$183.80/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 03:33 (138.9h) | Phase3: SKIPPED | SFT: Oct 26 08:18 (143.7h) | DPO (Final): Oct 26 08:52 (144.2h) [2026-10-20 08:38:49][INFO][training.pretrain_engine] Starting global step 351 (accumulating 32 micro-batches)... [2026-10-20 08:38:49][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 08:39:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3068 [2026-10-20 08:39:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4625 [2026-10-20 08:39:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8434 [2026-10-20 08:39:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5009 [2026-10-20 08:39:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5393 [2026-10-20 08:39:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2747 [2026-10-20 08:39:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1662 [2026-10-20 08:39:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4455 [2026-10-20 08:39:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7260 [2026-10-20 08:39:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6604 [2026-10-20 08:39:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7819 [2026-10-20 08:39:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2737 [2026-10-20 08:40:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4414 [2026-10-20 08:40:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4884 [2026-10-20 08:40:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2496 [2026-10-20 08:40:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4234 [2026-10-20 08:40:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8009 [2026-10-20 08:40:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4101 [2026-10-20 08:40:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3633 [2026-10-20 08:40:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9534 [2026-10-20 08:40:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8944 [2026-10-20 08:40:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6953 [2026-10-20 08:40:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7286 [2026-10-20 08:40:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1614 [2026-10-20 08:40:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7573 [2026-10-20 08:40:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9491 [2026-10-20 08:40:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.2330 [2026-10-20 08:40:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3695 [2026-10-20 08:41:09][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,513 | MFU: 1.8% [2026-10-20 08:41:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5017 [2026-10-20 08:41:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5424 [2026-10-20 08:41:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6798 [2026-10-20 08:41:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0715 [2026-10-20 08:41:13][INFO][training.pretrain_engine] [Ph2|Step 351] loss=6.4550 | ppl=635.86 | aux=0.02513 | lr=2.33e-04 | gnorm=1.297 | tok/s=29,245 | MFU=7.0% | step=143421ms | tokens=0.375B | mem=4.2/32.7GB | cost=$184.33/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 03:35 (138.9h) | Phase3: SKIPPED | SFT: Oct 26 08:20 (143.7h) | DPO (Final): Oct 26 08:55 (144.2h) [2026-10-20 08:41:16][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step351 | Tokens: 0.375B | Spent: $184.34 | Remaining: $215.66 [2026-10-20 08:41:16][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step351 [2026-10-20 08:41:16][INFO][training.pretrain_engine] Starting global step 352 (accumulating 32 micro-batches)... [2026-10-20 08:41:16][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 08:41:17][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 08:41:17][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 08:41:17][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 08:41:17][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step351 to Phase-Technologies/forge-500m-pretrain (path: phase2_step351)... [2026-10-20 08:41:17][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 08:41:27][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 08:41:27][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step351 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 08:41:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3314 [2026-10-20 08:41:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3570 [2026-10-20 08:41:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4388 [2026-10-20 08:41:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2715 [2026-10-20 08:41:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5293 [2026-10-20 08:41:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4929 [2026-10-20 08:41:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8657 [2026-10-20 08:41:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3655 [2026-10-20 08:42:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2219 [2026-10-20 08:42:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6539 [2026-10-20 08:42:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1751 [2026-10-20 08:42:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6037 [2026-10-20 08:42:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2739 [2026-10-20 08:42:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7218 [2026-10-20 08:42:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3101 [2026-10-20 08:42:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9030 [2026-10-20 08:42:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4887 [2026-10-20 08:42:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1818 [2026-10-20 08:42:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1908 [2026-10-20 08:42:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3961 [2026-10-20 08:43:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4718 [2026-10-20 08:43:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4408 [2026-10-20 08:43:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3881 [2026-10-20 08:43:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4397 [2026-10-20 08:43:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3372 [2026-10-20 08:43:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4756 [2026-10-20 08:43:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1993 [2026-10-20 08:43:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2780 [2026-10-20 08:43:36][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,509 | MFU: 1.8% [2026-10-20 08:43:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3347 [2026-10-20 08:43:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5264 [2026-10-20 08:43:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6663 [2026-10-20 08:43:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7486 [2026-10-20 08:43:41][INFO][training.pretrain_engine] [Ph2|Step 352] loss=6.4720 | ppl=646.81 | aux=0.02514 | lr=2.33e-04 | gnorm=2.375 | tok/s=29,243 | MFU=7.0% | step=143431ms | tokens=0.376B | mem=4.2/32.7GB | cost=$184.86/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 03:38 (138.9h) | Phase3: SKIPPED | SFT: Oct 26 08:23 (143.7h) | DPO (Final): Oct 26 08:57 (144.2h) [2026-10-20 08:43:41][INFO][training.pretrain_engine] Starting global step 353 (accumulating 32 micro-batches)... [2026-10-20 08:43:41][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 08:43:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2116 [2026-10-20 08:43:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6705 [2026-10-20 08:43:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3631 [2026-10-20 08:43:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6061 [2026-10-20 08:44:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2895 [2026-10-20 08:44:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2037 [2026-10-20 08:44:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2032 [2026-10-20 08:44:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4796 [2026-10-20 08:44:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4735 [2026-10-20 08:44:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4997 [2026-10-20 08:44:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8127 [2026-10-20 08:44:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5891 [2026-10-20 08:44:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7827 [2026-10-20 08:44:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6126 [2026-10-20 08:44:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7166 [2026-10-20 08:44:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8099 [2026-10-20 08:45:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3045 [2026-10-20 08:45:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1207 [2026-10-20 08:45:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0768 [2026-10-20 08:45:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1537 [2026-10-20 08:45:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3606 [2026-10-20 08:45:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7099 [2026-10-20 08:45:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2094 [2026-10-20 08:45:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9961 [2026-10-20 08:45:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1558 [2026-10-20 08:45:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2217 [2026-10-20 08:45:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1931 [2026-10-20 08:45:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1943 [2026-10-20 08:46:00][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,527 | MFU: 1.8% [2026-10-20 08:46:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8854 [2026-10-20 08:46:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2485 [2026-10-20 08:46:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9560 [2026-10-20 08:46:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1774 [2026-10-20 08:46:04][INFO][training.pretrain_engine] [Ph2|Step 353] loss=6.4146 | ppl=610.72 | aux=0.02517 | lr=2.34e-04 | gnorm=2.016 | tok/s=29,240 | MFU=7.0% | step=143445ms | tokens=0.377B | mem=4.2/32.7GB | cost=$185.39/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 03:40 (138.9h) | Phase3: SKIPPED | SFT: Oct 26 08:25 (143.7h) | DPO (Final): Oct 26 09:00 (144.2h) [2026-10-20 08:46:04][INFO][training.pretrain_engine] Starting global step 354 (accumulating 32 micro-batches)... [2026-10-20 08:46:04][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 08:46:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0782 [2026-10-20 08:46:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4490 [2026-10-20 08:46:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9453 [2026-10-20 08:46:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2922 [2026-10-20 08:46:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0895 [2026-10-20 08:46:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3969 [2026-10-20 08:46:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1853 [2026-10-20 08:46:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0859 [2026-10-20 08:46:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6168 [2026-10-20 08:46:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8549 [2026-10-20 08:46:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9938 [2026-10-20 08:46:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8514 [2026-10-20 08:47:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0518 [2026-10-20 08:47:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1859 [2026-10-20 08:47:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2537 [2026-10-20 08:47:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2778 [2026-10-20 08:47:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9031 [2026-10-20 08:47:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7740 [2026-10-20 08:47:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1637 [2026-10-20 08:47:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8828 [2026-10-20 08:47:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0374 [2026-10-20 08:47:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3408 [2026-10-20 08:47:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3895 [2026-10-20 08:47:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3114 [2026-10-20 08:48:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4014 [2026-10-20 08:48:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5189 [2026-10-20 08:48:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4894 [2026-10-20 08:48:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6726 [2026-10-20 08:48:23][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,547 | MFU: 1.8% [2026-10-20 08:48:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4763 [2026-10-20 08:48:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3221 [2026-10-20 08:48:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3603 [2026-10-20 08:48:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3715 [2026-10-20 08:48:28][INFO][training.pretrain_engine] [Ph2|Step 354] loss=6.3929 | ppl=597.57 | aux=0.02516 | lr=2.34e-04 | gnorm=1.250 | tok/s=29,239 | MFU=7.0% | step=143450ms | tokens=0.379B | mem=4.2/32.7GB | cost=$185.91/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 03:42 (138.9h) | Phase3: SKIPPED | SFT: Oct 26 08:27 (143.7h) | DPO (Final): Oct 26 09:02 (144.2h) [2026-10-20 08:48:28][INFO][training.pretrain_engine] Starting global step 355 (accumulating 32 micro-batches)... [2026-10-20 08:48:28][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 08:48:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0575 [2026-10-20 08:48:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9932 [2026-10-20 08:48:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.0351 [2026-10-20 08:48:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7920 [2026-10-20 08:49:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6795 [2026-10-20 08:49:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2406 [2026-10-20 08:49:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6555 [2026-10-20 08:49:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9151 [2026-10-20 08:49:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1635 [2026-10-20 08:49:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1799 [2026-10-20 08:49:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3034 [2026-10-20 08:49:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8619 [2026-10-20 08:49:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4299 [2026-10-20 08:49:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3637 [2026-10-20 08:49:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1959 [2026-10-20 08:49:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0140 [2026-10-20 08:49:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4792 [2026-10-20 08:49:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1139 [2026-10-20 08:49:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6872 [2026-10-20 08:49:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8538 [2026-10-20 08:50:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2290 [2026-10-20 08:50:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8254 [2026-10-20 08:50:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6664 [2026-10-20 08:50:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5462 [2026-10-20 08:50:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5795 [2026-10-20 08:50:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4533 [2026-10-20 08:50:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4869 [2026-10-20 08:50:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9123 [2026-10-20 08:50:47][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,531 | MFU: 1.8% [2026-10-20 08:50:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9274 [2026-10-20 08:50:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5354 [2026-10-20 08:50:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3703 [2026-10-20 08:50:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3382 [2026-10-20 08:50:52][INFO][training.pretrain_engine] [Ph2|Step 355] loss=6.4506 | ppl=633.10 | aux=0.02516 | lr=2.35e-04 | gnorm=2.031 | tok/s=29,238 | MFU=7.0% | step=143453ms | tokens=0.380B | mem=4.2/32.7GB | cost=$186.44/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 03:44 (138.9h) | Phase3: SKIPPED | SFT: Oct 26 08:30 (143.7h) | DPO (Final): Oct 26 09:04 (144.2h) [2026-10-20 08:50:52][INFO][training.pretrain_engine] Starting global step 356 (accumulating 32 micro-batches)... [2026-10-20 08:50:52][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 08:51:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2812 [2026-10-20 08:51:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2598 [2026-10-20 08:51:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3956 [2026-10-20 08:51:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5202 [2026-10-20 08:51:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4480 [2026-10-20 08:51:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3768 [2026-10-20 08:51:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4259 [2026-10-20 08:51:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5013 [2026-10-20 08:51:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6170 [2026-10-20 08:51:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1437 [2026-10-20 08:51:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8659 [2026-10-20 08:51:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6782 [2026-10-20 08:52:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9204 [2026-10-20 08:52:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1900 [2026-10-20 08:52:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6589 [2026-10-20 08:52:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4411 [2026-10-20 08:52:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6224 [2026-10-20 08:52:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7394 [2026-10-20 08:52:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6259 [2026-10-20 08:52:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7235 [2026-10-20 08:52:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9329 [2026-10-20 08:52:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0207 [2026-10-20 08:52:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3536 [2026-10-20 08:52:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9653 [2026-10-20 08:52:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3436 [2026-10-20 08:52:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3180 [2026-10-20 08:52:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6852 [2026-10-20 08:52:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4274 [2026-10-20 08:53:11][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,496 | MFU: 1.8% [2026-10-20 08:53:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4689 [2026-10-20 08:53:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5368 [2026-10-20 08:53:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5912 [2026-10-20 08:53:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4925 [2026-10-20 08:53:16][INFO][training.pretrain_engine] [Ph2|Step 356] loss=6.5370 | ppl=690.22 | aux=0.02515 | lr=2.36e-04 | gnorm=1.258 | tok/s=29,235 | MFU=7.0% | step=143468ms | tokens=0.381B | mem=4.2/32.7GB | cost=$186.97/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 03:47 (138.9h) | Phase3: SKIPPED | SFT: Oct 26 08:32 (143.7h) | DPO (Final): Oct 26 09:06 (144.2h) [2026-10-20 08:53:16][INFO][training.pretrain_engine] Starting global step 357 (accumulating 32 micro-batches)... [2026-10-20 08:53:16][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 08:53:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3308 [2026-10-20 08:53:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4492 [2026-10-20 08:53:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4916 [2026-10-20 08:53:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7855 [2026-10-20 08:53:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6051 [2026-10-20 08:53:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8874 [2026-10-20 08:53:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7751 [2026-10-20 08:53:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4124 [2026-10-20 08:54:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0574 [2026-10-20 08:54:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.4580 [2026-10-20 08:54:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4957 [2026-10-20 08:54:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7067 [2026-10-20 08:54:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8475 [2026-10-20 08:54:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8540 [2026-10-20 08:54:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3373 [2026-10-20 08:54:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2502 [2026-10-20 08:54:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3319 [2026-10-20 08:54:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6292 [2026-10-20 08:54:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8446 [2026-10-20 08:54:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3541 [2026-10-20 08:55:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6697 [2026-10-20 08:55:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7197 [2026-10-20 08:55:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1900 [2026-10-20 08:55:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1648 [2026-10-20 08:55:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4469 [2026-10-20 08:55:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3745 [2026-10-20 08:55:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4412 [2026-10-20 08:55:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2502 [2026-10-20 08:55:35][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,522 | MFU: 1.8% [2026-10-20 08:55:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3719 [2026-10-20 08:55:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2787 [2026-10-20 08:55:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4133 [2026-10-20 08:55:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2702 [2026-10-20 08:55:40][INFO][training.pretrain_engine] [Ph2|Step 357] loss=6.3762 | ppl=587.71 | aux=0.02518 | lr=2.36e-04 | gnorm=1.648 | tok/s=29,233 | MFU=7.0% | step=143480ms | tokens=0.382B | mem=4.2/32.7GB | cost=$187.49/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 03:50 (138.9h) | Phase3: SKIPPED | SFT: Oct 26 08:35 (143.7h) | DPO (Final): Oct 26 09:09 (144.2h) [2026-10-20 08:55:40][INFO][training.pretrain_engine] Starting global step 358 (accumulating 32 micro-batches)... [2026-10-20 08:55:40][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 08:55:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1535 [2026-10-20 08:55:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5131 [2026-10-20 08:55:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1856 [2026-10-20 08:55:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2950 [2026-10-20 08:56:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6378 [2026-10-20 08:56:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1626 [2026-10-20 08:56:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6227 [2026-10-20 08:56:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4775 [2026-10-20 08:56:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5454 [2026-10-20 08:56:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3628 [2026-10-20 08:56:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4202 [2026-10-20 08:56:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7915 [2026-10-20 08:56:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4772 [2026-10-20 08:56:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3773 [2026-10-20 08:56:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2972 [2026-10-20 08:56:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5424 [2026-10-20 08:57:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.2670 [2026-10-20 08:57:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7498 [2026-10-20 08:57:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9722 [2026-10-20 08:57:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9891 [2026-10-20 08:57:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8758 [2026-10-20 08:57:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0401 [2026-10-20 08:57:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1604 [2026-10-20 08:57:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6951 [2026-10-20 08:57:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2895 [2026-10-20 08:57:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9536 [2026-10-20 08:57:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6481 [2026-10-20 08:57:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4013 [2026-10-20 08:57:58][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,570 | MFU: 1.8% [2026-10-20 08:58:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5762 [2026-10-20 08:58:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8536 [2026-10-20 08:58:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5619 [2026-10-20 08:58:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3330 [2026-10-20 08:58:04][INFO][training.pretrain_engine] [Ph2|Step 358] loss=6.4932 | ppl=660.63 | aux=0.02514 | lr=2.37e-04 | gnorm=1.625 | tok/s=29,231 | MFU=7.0% | step=143488ms | tokens=0.383B | mem=4.2/32.7GB | cost=$188.02/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 03:52 (138.9h) | Phase3: SKIPPED | SFT: Oct 26 08:37 (143.7h) | DPO (Final): Oct 26 09:11 (144.2h) [2026-10-20 08:58:04][INFO][training.pretrain_engine] Starting global step 359 (accumulating 32 micro-batches)... [2026-10-20 08:58:04][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 08:58:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2688 [2026-10-20 08:58:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1702 [2026-10-20 08:58:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3597 [2026-10-20 08:58:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6366 [2026-10-20 08:58:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9333 [2026-10-20 08:58:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2208 [2026-10-20 08:58:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1981 [2026-10-20 08:58:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0961 [2026-10-20 08:58:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3738 [2026-10-20 08:58:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0931 [2026-10-20 08:58:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2612 [2026-10-20 08:58:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3033 [2026-10-20 08:59:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1729 [2026-10-20 08:59:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4224 [2026-10-20 08:59:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7859 [2026-10-20 08:59:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7855 [2026-10-20 08:59:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5846 [2026-10-20 08:59:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3204 [2026-10-20 08:59:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5829 [2026-10-20 08:59:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9285 [2026-10-20 08:59:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1539 [2026-10-20 08:59:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4570 [2026-10-20 08:59:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1918 [2026-10-20 08:59:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1469 [2026-10-20 09:00:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2999 [2026-10-20 09:00:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4706 [2026-10-20 09:00:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2264 [2026-10-20 09:00:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5070 [2026-10-20 09:00:22][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,575 | MFU: 1.8% [2026-10-20 09:00:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0590 [2026-10-20 09:00:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0033 [2026-10-20 09:00:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6665 [2026-10-20 09:00:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.1652 [2026-10-20 09:00:28][INFO][training.pretrain_engine] [Ph2|Step 359] loss=6.4704 | ppl=645.75 | aux=0.02514 | lr=2.37e-04 | gnorm=1.148 | tok/s=29,228 | MFU=7.0% | step=143501ms | tokens=0.384B | mem=4.2/32.7GB | cost=$188.55/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 03:54 (138.9h) | Phase3: SKIPPED | SFT: Oct 26 08:40 (143.7h) | DPO (Final): Oct 26 09:14 (144.2h) [2026-10-20 09:00:28][INFO][training.pretrain_engine] Starting global step 360 (accumulating 32 micro-batches)... [2026-10-20 09:00:28][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 09:00:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5709 [2026-10-20 09:00:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3979 [2026-10-20 09:00:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5782 [2026-10-20 09:00:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1370 [2026-10-20 09:01:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.3545 [2026-10-20 09:01:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9527 [2026-10-20 09:01:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7431 [2026-10-20 09:01:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9838 [2026-10-20 09:01:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8060 [2026-10-20 09:01:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6567 [2026-10-20 09:01:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5212 [2026-10-20 09:01:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5040 [2026-10-20 09:01:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9842 [2026-10-20 09:01:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9333 [2026-10-20 09:01:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7719 [2026-10-20 09:01:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2304 [2026-10-20 09:01:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2127 [2026-10-20 09:01:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7057 [2026-10-20 09:01:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4865 [2026-10-20 09:01:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8865 [2026-10-20 09:02:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6620 [2026-10-20 09:02:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3204 [2026-10-20 09:02:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3929 [2026-10-20 09:02:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9942 [2026-10-20 09:02:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1968 [2026-10-20 09:02:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1759 [2026-10-20 09:02:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0728 [2026-10-20 09:02:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4106 [2026-10-20 09:02:46][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,567 | MFU: 1.8% [2026-10-20 09:02:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8089 [2026-10-20 09:02:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2801 [2026-10-20 09:02:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1982 [2026-10-20 09:02:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3919 [2026-10-20 09:02:51][INFO][training.pretrain_engine] [Ph2|Step 360] loss=6.4146 | ppl=610.70 | aux=0.02513 | lr=2.38e-04 | gnorm=1.438 | tok/s=29,227 | MFU=7.0% | step=143506ms | tokens=0.385B | mem=4.2/32.7GB | cost=$189.07/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 03:57 (138.9h) | Phase3: SKIPPED | SFT: Oct 26 08:42 (143.7h) | DPO (Final): Oct 26 09:16 (144.2h) [2026-10-20 09:02:54][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step360 | Tokens: 0.385B | Spent: $189.08 | Remaining: $210.92 [2026-10-20 09:02:54][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step360 [2026-10-20 09:02:54][INFO][training.pretrain_engine] Starting global step 361 (accumulating 32 micro-batches)... [2026-10-20 09:02:54][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 09:02:54][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 09:02:54][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 09:02:54][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 09:02:54][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step360 to Phase-Technologies/forge-500m-pretrain (path: phase2_step360)... [2026-10-20 09:02:54][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 09:03:05][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 09:03:05][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step360 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 09:03:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4555 [2026-10-20 09:03:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8025 [2026-10-20 09:03:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7075 [2026-10-20 09:03:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3838 [2026-10-20 09:03:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3210 [2026-10-20 09:03:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3078 [2026-10-20 09:03:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4059 [2026-10-20 09:03:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4995 [2026-10-20 09:03:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4124 [2026-10-20 09:03:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4796 [2026-10-20 09:03:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9004 [2026-10-20 09:03:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0751 [2026-10-20 09:04:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2727 [2026-10-20 09:04:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2912 [2026-10-20 09:04:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1914 [2026-10-20 09:04:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2024 [2026-10-20 09:04:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4608 [2026-10-20 09:04:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1431 [2026-10-20 09:04:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3853 [2026-10-20 09:04:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5685 [2026-10-20 09:04:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7725 [2026-10-20 09:04:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2809 [2026-10-20 09:04:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0723 [2026-10-20 09:04:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4107 [2026-10-20 09:04:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2515 [2026-10-20 09:04:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3374 [2026-10-20 09:04:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3272 [2026-10-20 09:04:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4172 [2026-10-20 09:05:13][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,563 | MFU: 1.8% [2026-10-20 09:05:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9047 [2026-10-20 09:05:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4551 [2026-10-20 09:05:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0207 [2026-10-20 09:05:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5292 [2026-10-20 09:05:17][INFO][training.pretrain_engine] [Ph2|Step 361] loss=6.4776 | ppl=650.38 | aux=0.02514 | lr=2.38e-04 | gnorm=1.359 | tok/s=29,227 | MFU=7.0% | step=143509ms | tokens=0.386B | mem=4.2/32.7GB | cost=$189.61/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 03:59 (138.9h) | Phase3: SKIPPED | SFT: Oct 26 08:44 (143.6h) | DPO (Final): Oct 26 09:18 (144.2h) [2026-10-20 09:05:17][INFO][training.pretrain_engine] Starting global step 362 (accumulating 32 micro-batches)... [2026-10-20 09:05:17][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 09:05:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6819 [2026-10-20 09:05:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3269 [2026-10-20 09:05:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9068 [2026-10-20 09:05:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6496 [2026-10-20 09:05:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8890 [2026-10-20 09:05:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6783 [2026-10-20 09:05:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4984 [2026-10-20 09:05:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6866 [2026-10-20 09:06:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3274 [2026-10-20 09:06:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2808 [2026-10-20 09:06:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5057 [2026-10-20 09:06:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2586 [2026-10-20 09:06:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7530 [2026-10-20 09:06:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3409 [2026-10-20 09:06:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9531 [2026-10-20 09:06:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1509 [2026-10-20 09:06:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7447 [2026-10-20 09:06:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8591 [2026-10-20 09:06:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6229 [2026-10-20 09:06:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6478 [2026-10-20 09:07:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1874 [2026-10-20 09:07:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2608 [2026-10-20 09:07:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2163 [2026-10-20 09:07:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3128 [2026-10-20 09:07:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3532 [2026-10-20 09:07:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1430 [2026-10-20 09:07:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1992 [2026-10-20 09:07:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3856 [2026-10-20 09:07:36][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,536 | MFU: 1.8% [2026-10-20 09:07:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6603 [2026-10-20 09:07:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9679 [2026-10-20 09:07:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3741 [2026-10-20 09:07:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4534 [2026-10-20 09:07:41][INFO][training.pretrain_engine] [Ph2|Step 362] loss=6.3876 | ppl=594.41 | aux=0.02514 | lr=2.39e-04 | gnorm=0.965 | tok/s=29,226 | MFU=7.0% | step=143515ms | tokens=0.387B | mem=4.2/32.7GB | cost=$190.13/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 04:01 (138.9h) | Phase3: SKIPPED | SFT: Oct 26 08:46 (143.6h) | DPO (Final): Oct 26 09:20 (144.2h) [2026-10-20 09:07:41][INFO][training.pretrain_engine] Starting global step 363 (accumulating 32 micro-batches)... [2026-10-20 09:07:41][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 09:07:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7252 [2026-10-20 09:07:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4566 [2026-10-20 09:07:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7076 [2026-10-20 09:07:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7925 [2026-10-20 09:08:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3904 [2026-10-20 09:08:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8381 [2026-10-20 09:08:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2380 [2026-10-20 09:08:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8629 [2026-10-20 09:08:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7922 [2026-10-20 09:08:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9421 [2026-10-20 09:08:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6104 [2026-10-20 09:08:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2665 [2026-10-20 09:08:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5870 [2026-10-20 09:08:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5473 [2026-10-20 09:08:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2194 [2026-10-20 09:08:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3551 [2026-10-20 09:09:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1953 [2026-10-20 09:09:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1894 [2026-10-20 09:09:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5828 [2026-10-20 09:09:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2748 [2026-10-20 09:09:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4507 [2026-10-20 09:09:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4262 [2026-10-20 09:09:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3211 [2026-10-20 09:09:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3584 [2026-10-20 09:09:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8465 [2026-10-20 09:09:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9310 [2026-10-20 09:09:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3197 [2026-10-20 09:09:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4767 [2026-10-20 09:10:00][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,536 | MFU: 1.8% [2026-10-20 09:10:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3861 [2026-10-20 09:10:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9104 [2026-10-20 09:10:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3106 [2026-10-20 09:10:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3101 [2026-10-20 09:10:05][INFO][training.pretrain_engine] [Ph2|Step 363] loss=6.3696 | ppl=583.81 | aux=0.02512 | lr=2.40e-04 | gnorm=1.109 | tok/s=29,224 | MFU=7.0% | step=143520ms | tokens=0.388B | mem=4.2/32.7GB | cost=$190.66/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 04:03 (138.9h) | Phase3: SKIPPED | SFT: Oct 26 08:48 (143.6h) | DPO (Final): Oct 26 09:22 (144.2h) [2026-10-20 09:10:05][INFO][training.pretrain_engine] Starting global step 364 (accumulating 32 micro-batches)... [2026-10-20 09:10:05][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 09:10:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0619 [2026-10-20 09:10:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1609 [2026-10-20 09:10:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1354 [2026-10-20 09:10:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2807 [2026-10-20 09:10:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5226 [2026-10-20 09:10:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6244 [2026-10-20 09:10:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1629 [2026-10-20 09:10:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4454 [2026-10-20 09:10:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0080 [2026-10-20 09:10:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3791 [2026-10-20 09:10:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3240 [2026-10-20 09:10:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0949 [2026-10-20 09:11:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6795 [2026-10-20 09:11:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1054 [2026-10-20 09:11:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5122 [2026-10-20 09:11:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2638 [2026-10-20 09:11:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6988 [2026-10-20 09:11:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8088 [2026-10-20 09:11:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0961 [2026-10-20 09:11:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4828 [2026-10-20 09:11:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0273 [2026-10-20 09:11:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2374 [2026-10-20 09:11:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0570 [2026-10-20 09:11:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1845 [2026-10-20 09:12:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1280 [2026-10-20 09:12:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8720 [2026-10-20 09:12:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8627 [2026-10-20 09:12:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9706 [2026-10-20 09:12:24][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,524 | MFU: 1.8% [2026-10-20 09:12:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7683 [2026-10-20 09:12:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4959 [2026-10-20 09:12:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4411 [2026-10-20 09:12:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4080 [2026-10-20 09:12:29][INFO][training.pretrain_engine] [Ph2|Step 364] loss=6.4156 | ppl=611.28 | aux=0.02514 | lr=2.40e-04 | gnorm=1.094 | tok/s=29,222 | MFU=7.0% | step=143532ms | tokens=0.389B | mem=4.2/32.7GB | cost=$191.18/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 04:05 (138.9h) | Phase3: SKIPPED | SFT: Oct 26 08:50 (143.6h) | DPO (Final): Oct 26 09:25 (144.2h) [2026-10-20 09:12:29][INFO][training.pretrain_engine] Starting global step 365 (accumulating 32 micro-batches)... [2026-10-20 09:12:29][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 09:12:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8065 [2026-10-20 09:12:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1968 [2026-10-20 09:12:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6722 [2026-10-20 09:12:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.1181 [2026-10-20 09:13:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4403 [2026-10-20 09:13:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9672 [2026-10-20 09:13:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9276 [2026-10-20 09:13:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3205 [2026-10-20 09:13:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7038 [2026-10-20 09:13:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6897 [2026-10-20 09:13:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7669 [2026-10-20 09:13:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4844 [2026-10-20 09:13:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5224 [2026-10-20 09:13:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0267 [2026-10-20 09:13:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0547 [2026-10-20 09:13:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3864 [2026-10-20 09:13:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7884 [2026-10-20 09:13:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2945 [2026-10-20 09:13:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5482 [2026-10-20 09:13:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8968 [2026-10-20 09:14:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7099 [2026-10-20 09:14:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4060 [2026-10-20 09:14:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5549 [2026-10-20 09:14:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7608 [2026-10-20 09:14:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4961 [2026-10-20 09:14:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4785 [2026-10-20 09:14:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1078 [2026-10-20 09:14:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2542 [2026-10-20 09:14:48][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,538 | MFU: 1.8% [2026-10-20 09:14:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6905 [2026-10-20 09:14:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5685 [2026-10-20 09:14:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5413 [2026-10-20 09:14:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3630 [2026-10-20 09:14:52][INFO][training.pretrain_engine] [Ph2|Step 365] loss=6.4077 | ppl=606.48 | aux=0.02512 | lr=2.41e-04 | gnorm=1.109 | tok/s=29,221 | MFU=7.0% | step=143539ms | tokens=0.390B | mem=4.2/32.7GB | cost=$191.71/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 04:07 (138.9h) | Phase3: SKIPPED | SFT: Oct 26 08:53 (143.6h) | DPO (Final): Oct 26 09:27 (144.2h) [2026-10-20 09:14:52][INFO][training.pretrain_engine] Starting global step 366 (accumulating 32 micro-batches)... [2026-10-20 09:14:52][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 09:15:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2299 [2026-10-20 09:15:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3436 [2026-10-20 09:15:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3467 [2026-10-20 09:15:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9998 [2026-10-20 09:15:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8071 [2026-10-20 09:15:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5531 [2026-10-20 09:15:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1674 [2026-10-20 09:15:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8429 [2026-10-20 09:15:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3912 [2026-10-20 09:15:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8266 [2026-10-20 09:15:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7175 [2026-10-20 09:15:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8750 [2026-10-20 09:16:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4643 [2026-10-20 09:16:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3157 [2026-10-20 09:16:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0951 [2026-10-20 09:16:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7416 [2026-10-20 09:16:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6266 [2026-10-20 09:16:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7570 [2026-10-20 09:16:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1585 [2026-10-20 09:16:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0697 [2026-10-20 09:16:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6002 [2026-10-20 09:16:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5506 [2026-10-20 09:16:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0046 [2026-10-20 09:16:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3176 [2026-10-20 09:16:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5965 [2026-10-20 09:16:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0917 [2026-10-20 09:16:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4446 [2026-10-20 09:16:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1157 [2026-10-20 09:17:12][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,529 | MFU: 1.8% [2026-10-20 09:17:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9521 [2026-10-20 09:17:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1174 [2026-10-20 09:17:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2868 [2026-10-20 09:17:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1675 [2026-10-20 09:17:16][INFO][training.pretrain_engine] [Ph2|Step 366] loss=6.5273 | ppl=683.52 | aux=0.02515 | lr=2.41e-04 | gnorm=1.195 | tok/s=29,219 | MFU=7.0% | step=143547ms | tokens=0.391B | mem=4.2/32.7GB | cost=$192.23/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 04:10 (138.9h) | Phase3: SKIPPED | SFT: Oct 26 08:55 (143.6h) | DPO (Final): Oct 26 09:29 (144.2h) [2026-10-20 09:17:16][INFO][training.pretrain_engine] Starting global step 367 (accumulating 32 micro-batches)... [2026-10-20 09:17:16][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 09:17:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6570 [2026-10-20 09:17:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3229 [2026-10-20 09:17:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1186 [2026-10-20 09:17:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6367 [2026-10-20 09:17:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2120 [2026-10-20 09:17:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4198 [2026-10-20 09:17:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3067 [2026-10-20 09:17:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4162 [2026-10-20 09:18:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5498 [2026-10-20 09:18:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.0429 [2026-10-20 09:18:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4177 [2026-10-20 09:18:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6717 [2026-10-20 09:18:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5524 [2026-10-20 09:18:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4484 [2026-10-20 09:18:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4828 [2026-10-20 09:18:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1273 [2026-10-20 09:18:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2874 [2026-10-20 09:18:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6843 [2026-10-20 09:18:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0064 [2026-10-20 09:18:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4954 [2026-10-20 09:19:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4409 [2026-10-20 09:19:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9378 [2026-10-20 09:19:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1186 [2026-10-20 09:19:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9487 [2026-10-20 09:19:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1773 [2026-10-20 09:19:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5831 [2026-10-20 09:19:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1856 [2026-10-20 09:19:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3809 [2026-10-20 09:19:36][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,519 | MFU: 1.8% [2026-10-20 09:19:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2643 [2026-10-20 09:19:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8586 [2026-10-20 09:19:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0344 [2026-10-20 09:19:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5979 [2026-10-20 09:19:40][INFO][training.pretrain_engine] [Ph2|Step 367] loss=6.5107 | ppl=672.27 | aux=0.02511 | lr=2.42e-04 | gnorm=0.703 | tok/s=29,217 | MFU=7.0% | step=143555ms | tokens=0.392B | mem=4.2/32.7GB | cost=$192.76/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 04:12 (138.9h) | Phase3: SKIPPED | SFT: Oct 26 08:57 (143.6h) | DPO (Final): Oct 26 09:31 (144.2h) [2026-10-20 09:19:40][INFO][training.pretrain_engine] Starting global step 368 (accumulating 32 micro-batches)... [2026-10-20 09:19:40][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 09:19:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9344 [2026-10-20 09:19:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7234 [2026-10-20 09:19:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3020 [2026-10-20 09:19:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9198 [2026-10-20 09:20:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1577 [2026-10-20 09:20:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4192 [2026-10-20 09:20:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4547 [2026-10-20 09:20:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5026 [2026-10-20 09:20:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8778 [2026-10-20 09:20:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6760 [2026-10-20 09:20:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3572 [2026-10-20 09:20:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5605 [2026-10-20 09:20:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8308 [2026-10-20 09:20:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4668 [2026-10-20 09:20:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4564 [2026-10-20 09:20:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4700 [2026-10-20 09:21:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0396 [2026-10-20 09:21:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3589 [2026-10-20 09:21:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1355 [2026-10-20 09:21:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0610 [2026-10-20 09:21:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6503 [2026-10-20 09:21:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5819 [2026-10-20 09:21:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4573 [2026-10-20 09:21:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6054 [2026-10-20 09:21:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7094 [2026-10-20 09:21:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3986 [2026-10-20 09:21:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5998 [2026-10-20 09:21:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2360 [2026-10-20 09:22:00][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,518 | MFU: 1.8% [2026-10-20 09:22:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5497 [2026-10-20 09:22:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2425 [2026-10-20 09:22:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.2065 [2026-10-20 09:22:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7932 [2026-10-20 09:22:04][INFO][training.pretrain_engine] [Ph2|Step 368] loss=6.3992 | ppl=601.38 | aux=0.02511 | lr=2.42e-04 | gnorm=1.094 | tok/s=29,216 | MFU=7.0% | step=143564ms | tokens=0.393B | mem=4.2/32.7GB | cost=$193.29/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 04:14 (138.9h) | Phase3: SKIPPED | SFT: Oct 26 09:00 (143.6h) | DPO (Final): Oct 26 09:34 (144.2h) [2026-10-20 09:22:04][INFO][training.pretrain_engine] Starting global step 369 (accumulating 32 micro-batches)... [2026-10-20 09:22:04][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 09:22:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3292 [2026-10-20 09:22:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5545 [2026-10-20 09:22:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2765 [2026-10-20 09:22:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6673 [2026-10-20 09:22:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5081 [2026-10-20 09:22:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1662 [2026-10-20 09:22:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4839 [2026-10-20 09:22:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5810 [2026-10-20 09:22:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3901 [2026-10-20 09:22:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3790 [2026-10-20 09:22:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5070 [2026-10-20 09:22:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1572 [2026-10-20 09:23:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9039 [2026-10-20 09:23:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3573 [2026-10-20 09:23:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3798 [2026-10-20 09:23:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3985 [2026-10-20 09:23:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5784 [2026-10-20 09:23:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8036 [2026-10-20 09:23:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0837 [2026-10-20 09:23:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2646 [2026-10-20 09:23:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4887 [2026-10-20 09:23:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2144 [2026-10-20 09:23:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2195 [2026-10-20 09:23:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9281 [2026-10-20 09:24:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2375 [2026-10-20 09:24:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2938 [2026-10-20 09:24:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1390 [2026-10-20 09:24:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1063 [2026-10-20 09:24:23][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,531 | MFU: 1.8% [2026-10-20 09:24:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3851 [2026-10-20 09:24:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5446 [2026-10-20 09:24:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7221 [2026-10-20 09:24:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2918 [2026-10-20 09:24:28][INFO][training.pretrain_engine] [Ph2|Step 369] loss=6.2682 | ppl=527.54 | aux=0.02514 | lr=2.43e-04 | gnorm=1.336 | tok/s=29,214 | MFU=7.0% | step=143573ms | tokens=0.394B | mem=4.2/32.7GB | cost=$193.81/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 04:17 (138.9h) | Phase3: SKIPPED | SFT: Oct 26 09:02 (143.6h) | DPO (Final): Oct 26 09:36 (144.2h) [2026-10-20 09:24:31][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step369 | Tokens: 0.394B | Spent: $193.82 | Remaining: $206.18 [2026-10-20 09:24:31][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step369 [2026-10-20 09:24:31][INFO][training.pretrain_engine] Starting global step 370 (accumulating 32 micro-batches)... [2026-10-20 09:24:31][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 09:24:31][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 09:24:31][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 09:24:31][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 09:24:31][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step369 to Phase-Technologies/forge-500m-pretrain (path: phase2_step369)... [2026-10-20 09:24:31][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 09:24:41][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 09:24:41][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step369 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 09:24:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4104 [2026-10-20 09:24:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7251 [2026-10-20 09:24:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1529 [2026-10-20 09:24:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6353 [2026-10-20 09:25:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7531 [2026-10-20 09:25:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2480 [2026-10-20 09:25:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4443 [2026-10-20 09:25:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1423 [2026-10-20 09:25:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9377 [2026-10-20 09:25:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3873 [2026-10-20 09:25:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1775 [2026-10-20 09:25:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1314 [2026-10-20 09:25:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9351 [2026-10-20 09:25:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5297 [2026-10-20 09:25:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0107 [2026-10-20 09:25:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6285 [2026-10-20 09:25:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3064 [2026-10-20 09:25:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6452 [2026-10-20 09:25:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4756 [2026-10-20 09:26:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4910 [2026-10-20 09:26:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2332 [2026-10-20 09:26:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5375 [2026-10-20 09:26:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5534 [2026-10-20 09:26:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7253 [2026-10-20 09:26:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1087 [2026-10-20 09:26:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4502 [2026-10-20 09:26:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4652 [2026-10-20 09:26:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2906 [2026-10-20 09:26:50][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,555 | MFU: 1.8% [2026-10-20 09:26:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5102 [2026-10-20 09:26:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3115 [2026-10-20 09:26:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3386 [2026-10-20 09:26:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7255 [2026-10-20 09:26:54][INFO][training.pretrain_engine] [Ph2|Step 370] loss=6.4044 | ppl=604.48 | aux=0.02513 | lr=2.43e-04 | gnorm=0.891 | tok/s=29,213 | MFU=7.0% | step=143577ms | tokens=0.395B | mem=4.2/32.7GB | cost=$194.35/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 04:19 (138.9h) | Phase3: SKIPPED | SFT: Oct 26 09:04 (143.6h) | DPO (Final): Oct 26 09:38 (144.2h) [2026-10-20 09:26:54][INFO][training.pretrain_engine] Starting global step 371 (accumulating 32 micro-batches)... [2026-10-20 09:26:54][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 09:27:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0963 [2026-10-20 09:27:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0836 [2026-10-20 09:27:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1732 [2026-10-20 09:27:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3600 [2026-10-20 09:27:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6755 [2026-10-20 09:27:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4933 [2026-10-20 09:27:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4199 [2026-10-20 09:27:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3232 [2026-10-20 09:27:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5961 [2026-10-20 09:27:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7179 [2026-10-20 09:27:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1423 [2026-10-20 09:27:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4008 [2026-10-20 09:28:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6715 [2026-10-20 09:28:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9856 [2026-10-20 09:28:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4478 [2026-10-20 09:28:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5690 [2026-10-20 09:28:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7087 [2026-10-20 09:28:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7082 [2026-10-20 09:28:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3891 [2026-10-20 09:28:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1326 [2026-10-20 09:28:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2034 [2026-10-20 09:28:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3929 [2026-10-20 09:28:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4145 [2026-10-20 09:28:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4094 [2026-10-20 09:29:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3839 [2026-10-20 09:29:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1815 [2026-10-20 09:29:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9871 [2026-10-20 09:29:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3875 [2026-10-20 09:29:13][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,562 | MFU: 1.8% [2026-10-20 09:29:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1642 [2026-10-20 09:29:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8152 [2026-10-20 09:29:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5275 [2026-10-20 09:29:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.9320 [2026-10-20 09:29:18][INFO][training.pretrain_engine] [Ph2|Step 371] loss=6.4311 | ppl=620.87 | aux=0.02515 | lr=2.44e-04 | gnorm=1.672 | tok/s=29,212 | MFU=7.0% | step=143580ms | tokens=0.396B | mem=4.2/32.7GB | cost=$194.87/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 04:21 (138.9h) | Phase3: SKIPPED | SFT: Oct 26 09:06 (143.6h) | DPO (Final): Oct 26 09:40 (144.2h) [2026-10-20 09:29:18][INFO][training.pretrain_engine] Starting global step 372 (accumulating 32 micro-batches)... [2026-10-20 09:29:18][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 09:29:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5601 [2026-10-20 09:29:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2638 [2026-10-20 09:29:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7963 [2026-10-20 09:29:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8839 [2026-10-20 09:29:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4076 [2026-10-20 09:29:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.3451 [2026-10-20 09:29:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1673 [2026-10-20 09:29:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6246 [2026-10-20 09:30:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2291 [2026-10-20 09:30:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3454 [2026-10-20 09:30:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5911 [2026-10-20 09:30:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1252 [2026-10-20 09:30:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2974 [2026-10-20 09:30:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1022 [2026-10-20 09:30:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7608 [2026-10-20 09:30:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9084 [2026-10-20 09:30:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0808 [2026-10-20 09:30:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4447 [2026-10-20 09:30:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6845 [2026-10-20 09:30:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7992 [2026-10-20 09:31:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0336 [2026-10-20 09:31:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9968 [2026-10-20 09:31:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2776 [2026-10-20 09:31:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4896 [2026-10-20 09:31:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2041 [2026-10-20 09:31:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5549 [2026-10-20 09:31:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3222 [2026-10-20 09:31:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0185 [2026-10-20 09:31:37][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,518 | MFU: 1.8% [2026-10-20 09:31:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2894 [2026-10-20 09:31:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5129 [2026-10-20 09:31:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6710 [2026-10-20 09:31:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2142 [2026-10-20 09:31:42][INFO][training.pretrain_engine] [Ph2|Step 372] loss=6.2355 | ppl=510.53 | aux=0.02512 | lr=2.45e-04 | gnorm=1.055 | tok/s=29,209 | MFU=7.0% | step=143596ms | tokens=0.397B | mem=4.2/32.7GB | cost=$195.40/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 04:23 (138.9h) | Phase3: SKIPPED | SFT: Oct 26 09:09 (143.6h) | DPO (Final): Oct 26 09:43 (144.2h) [2026-10-20 09:31:42][INFO][training.pretrain_engine] Starting global step 373 (accumulating 32 micro-batches)... [2026-10-20 09:31:42][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 09:32:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1956 [2026-10-20 09:32:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8342 [2026-10-20 09:32:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4495 [2026-10-20 09:32:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2530 [2026-10-20 09:32:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2568 [2026-10-20 09:32:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0242 [2026-10-20 09:32:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2316 [2026-10-20 09:32:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2731 [2026-10-20 09:32:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4980 [2026-10-20 09:32:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4730 [2026-10-20 09:32:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6331 [2026-10-20 09:32:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2439 [2026-10-20 09:32:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6778 [2026-10-20 09:32:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3886 [2026-10-20 09:32:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1947 [2026-10-20 09:32:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4087 [2026-10-20 09:33:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3166 [2026-10-20 09:33:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1952 [2026-10-20 09:33:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0694 [2026-10-20 09:33:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4165 [2026-10-20 09:33:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3034 [2026-10-20 09:33:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1516 [2026-10-20 09:33:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0888 [2026-10-20 09:33:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2094 [2026-10-20 09:33:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4329 [2026-10-20 09:33:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2640 [2026-10-20 09:33:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7172 [2026-10-20 09:33:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7596 [2026-10-20 09:34:01][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,520 | MFU: 1.8% [2026-10-20 09:34:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1742 [2026-10-20 09:34:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3286 [2026-10-20 09:34:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8104 [2026-10-20 09:34:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9682 [2026-10-20 09:34:06][INFO][training.pretrain_engine] [Ph2|Step 373] loss=6.4460 | ppl=630.17 | aux=0.02516 | lr=2.45e-04 | gnorm=2.406 | tok/s=29,207 | MFU=7.0% | step=143606ms | tokens=0.398B | mem=4.2/32.7GB | cost=$195.92/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 04:26 (138.9h) | Phase3: SKIPPED | SFT: Oct 26 09:11 (143.6h) | DPO (Final): Oct 26 09:45 (144.2h) [2026-10-20 09:34:06][INFO][training.pretrain_engine] Starting global step 374 (accumulating 32 micro-batches)... [2026-10-20 09:34:06][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 09:34:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2230 [2026-10-20 09:34:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1207 [2026-10-20 09:34:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4526 [2026-10-20 09:34:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3322 [2026-10-20 09:34:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3573 [2026-10-20 09:34:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0685 [2026-10-20 09:34:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3053 [2026-10-20 09:34:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4924 [2026-10-20 09:34:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7835 [2026-10-20 09:35:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5457 [2026-10-20 09:35:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8601 [2026-10-20 09:35:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7603 [2026-10-20 09:35:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2600 [2026-10-20 09:35:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5254 [2026-10-20 09:35:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6912 [2026-10-20 09:35:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4634 [2026-10-20 09:35:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1893 [2026-10-20 09:35:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6484 [2026-10-20 09:35:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1641 [2026-10-20 09:35:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8529 [2026-10-20 09:35:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3565 [2026-10-20 09:35:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6616 [2026-10-20 09:35:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1693 [2026-10-20 09:35:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0941 [2026-10-20 09:36:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6173 [2026-10-20 09:36:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6532 [2026-10-20 09:36:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2659 [2026-10-20 09:36:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9002 [2026-10-20 09:36:25][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,517 | MFU: 1.8% [2026-10-20 09:36:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3787 [2026-10-20 09:36:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9495 [2026-10-20 09:36:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7074 [2026-10-20 09:36:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5177 [2026-10-20 09:36:30][INFO][training.pretrain_engine] [Ph2|Step 374] loss=6.4233 | ppl=616.05 | aux=0.02517 | lr=2.46e-04 | gnorm=1.906 | tok/s=29,205 | MFU=7.0% | step=143615ms | tokens=0.400B | mem=4.2/32.7GB | cost=$196.45/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 04:28 (138.9h) | Phase3: SKIPPED | SFT: Oct 26 09:13 (143.6h) | DPO (Final): Oct 26 09:48 (144.2h) [2026-10-20 09:36:30][INFO][training.pretrain_engine] Starting global step 375 (accumulating 32 micro-batches)... [2026-10-20 09:36:30][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 09:36:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1757 [2026-10-20 09:36:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7742 [2026-10-20 09:36:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3982 [2026-10-20 09:36:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0597 [2026-10-20 09:37:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9739 [2026-10-20 09:37:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2067 [2026-10-20 09:37:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1967 [2026-10-20 09:37:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9392 [2026-10-20 09:37:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6902 [2026-10-20 09:37:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8661 [2026-10-20 09:37:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9516 [2026-10-20 09:37:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3180 [2026-10-20 09:37:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1911 [2026-10-20 09:37:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4885 [2026-10-20 09:37:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6620 [2026-10-20 09:37:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4479 [2026-10-20 09:37:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4639 [2026-10-20 09:38:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8906 [2026-10-20 09:38:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0846 [2026-10-20 09:38:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1854 [2026-10-20 09:38:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2862 [2026-10-20 09:38:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2678 [2026-10-20 09:38:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3314 [2026-10-20 09:38:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2602 [2026-10-20 09:38:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3680 [2026-10-20 09:38:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5128 [2026-10-20 09:38:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6134 [2026-10-20 09:38:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3690 [2026-10-20 09:38:50][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,513 | MFU: 1.8% [2026-10-20 09:38:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2833 [2026-10-20 09:38:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3094 [2026-10-20 09:38:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2247 [2026-10-20 09:38:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9011 [2026-10-20 09:38:54][INFO][training.pretrain_engine] [Ph2|Step 375] loss=6.3241 | ppl=557.84 | aux=0.02514 | lr=2.46e-04 | gnorm=1.531 | tok/s=29,203 | MFU=7.0% | step=143623ms | tokens=0.401B | mem=4.2/32.7GB | cost=$196.98/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 04:30 (138.9h) | Phase3: SKIPPED | SFT: Oct 26 09:16 (143.6h) | DPO (Final): Oct 26 09:50 (144.2h) [2026-10-20 09:38:54][INFO][training.pretrain_engine] Starting global step 376 (accumulating 32 micro-batches)... [2026-10-20 09:38:54][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 09:39:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5845 [2026-10-20 09:39:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2064 [2026-10-20 09:39:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8698 [2026-10-20 09:39:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8239 [2026-10-20 09:39:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7150 [2026-10-20 09:39:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4626 [2026-10-20 09:39:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7724 [2026-10-20 09:39:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3141 [2026-10-20 09:39:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8213 [2026-10-20 09:39:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0141 [2026-10-20 09:39:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1658 [2026-10-20 09:39:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9496 [2026-10-20 09:40:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6576 [2026-10-20 09:40:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5899 [2026-10-20 09:40:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7458 [2026-10-20 09:40:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0659 [2026-10-20 09:40:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3115 [2026-10-20 09:40:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3958 [2026-10-20 09:40:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8507 [2026-10-20 09:40:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4963 [2026-10-20 09:40:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0640 [2026-10-20 09:40:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1879 [2026-10-20 09:40:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3089 [2026-10-20 09:40:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1837 [2026-10-20 09:40:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3090 [2026-10-20 09:40:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0968 [2026-10-20 09:40:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6366 [2026-10-20 09:41:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9388 [2026-10-20 09:41:13][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,526 | MFU: 1.8% [2026-10-20 09:41:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3563 [2026-10-20 09:41:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5063 [2026-10-20 09:41:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3394 [2026-10-20 09:41:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1591 [2026-10-20 09:41:18][INFO][training.pretrain_engine] [Ph2|Step 376] loss=6.2215 | ppl=503.46 | aux=0.02514 | lr=2.47e-04 | gnorm=1.891 | tok/s=29,202 | MFU=7.0% | step=143631ms | tokens=0.402B | mem=4.2/32.7GB | cost=$197.50/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 04:33 (138.9h) | Phase3: SKIPPED | SFT: Oct 26 09:18 (143.6h) | DPO (Final): Oct 26 09:52 (144.2h) [2026-10-20 09:41:18][INFO][training.pretrain_engine] Starting global step 377 (accumulating 32 micro-batches)... [2026-10-20 09:41:18][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 09:41:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4793 [2026-10-20 09:41:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1499 [2026-10-20 09:41:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1728 [2026-10-20 09:41:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2862 [2026-10-20 09:41:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0765 [2026-10-20 09:41:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5790 [2026-10-20 09:41:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5697 [2026-10-20 09:41:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3675 [2026-10-20 09:42:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0614 [2026-10-20 09:42:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0038 [2026-10-20 09:42:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1037 [2026-10-20 09:42:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1549 [2026-10-20 09:42:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2023 [2026-10-20 09:42:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1435 [2026-10-20 09:42:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2675 [2026-10-20 09:42:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5960 [2026-10-20 09:42:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2398 [2026-10-20 09:42:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3254 [2026-10-20 09:42:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5808 [2026-10-20 09:42:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7472 [2026-10-20 09:43:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3402 [2026-10-20 09:43:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4876 [2026-10-20 09:43:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4821 [2026-10-20 09:43:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3748 [2026-10-20 09:43:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8227 [2026-10-20 09:43:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0245 [2026-10-20 09:43:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8100 [2026-10-20 09:43:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4656 [2026-10-20 09:43:37][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,544 | MFU: 1.8% [2026-10-20 09:43:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2463 [2026-10-20 09:43:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9546 [2026-10-20 09:43:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0793 [2026-10-20 09:43:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3782 [2026-10-20 09:43:42][INFO][training.pretrain_engine] [Ph2|Step 377] loss=6.4276 | ppl=618.66 | aux=0.02512 | lr=2.47e-04 | gnorm=1.242 | tok/s=29,201 | MFU=7.0% | step=143634ms | tokens=0.403B | mem=4.2/32.7GB | cost=$198.03/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 04:35 (138.9h) | Phase3: SKIPPED | SFT: Oct 26 09:20 (143.6h) | DPO (Final): Oct 26 09:54 (144.2h) [2026-10-20 09:43:42][INFO][training.pretrain_engine] Starting global step 378 (accumulating 32 micro-batches)... [2026-10-20 09:43:42][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 09:43:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8686 [2026-10-20 09:43:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7662 [2026-10-20 09:43:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7617 [2026-10-20 09:44:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9782 [2026-10-20 09:44:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1697 [2026-10-20 09:44:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4247 [2026-10-20 09:44:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3424 [2026-10-20 09:44:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4103 [2026-10-20 09:44:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3360 [2026-10-20 09:44:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8891 [2026-10-20 09:44:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5861 [2026-10-20 09:44:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9471 [2026-10-20 09:44:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2915 [2026-10-20 09:44:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2291 [2026-10-20 09:44:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5490 [2026-10-20 09:44:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0892 [2026-10-20 09:45:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1676 [2026-10-20 09:45:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8625 [2026-10-20 09:45:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1591 [2026-10-20 09:45:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0498 [2026-10-20 09:45:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2578 [2026-10-20 09:45:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4512 [2026-10-20 09:45:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2633 [2026-10-20 09:45:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9531 [2026-10-20 09:45:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2552 [2026-10-20 09:45:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2118 [2026-10-20 09:45:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3050 [2026-10-20 09:45:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5743 [2026-10-20 09:46:01][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,528 | MFU: 1.8% [2026-10-20 09:46:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5478 [2026-10-20 09:46:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7966 [2026-10-20 09:46:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6728 [2026-10-20 09:46:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8022 [2026-10-20 09:46:05][INFO][training.pretrain_engine] [Ph2|Step 378] loss=6.3473 | ppl=570.93 | aux=0.02515 | lr=2.48e-04 | gnorm=1.852 | tok/s=29,201 | MFU=7.0% | step=143636ms | tokens=0.404B | mem=4.2/32.7GB | cost=$198.55/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 04:37 (138.8h) | Phase3: SKIPPED | SFT: Oct 26 09:22 (143.6h) | DPO (Final): Oct 26 09:56 (144.2h) [2026-10-20 09:46:09][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step378 | Tokens: 0.404B | Spent: $198.57 | Remaining: $201.43 [2026-10-20 09:46:09][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step378 [2026-10-20 09:46:09][INFO][training.pretrain_engine] Starting global step 379 (accumulating 32 micro-batches)... [2026-10-20 09:46:09][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 09:46:09][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 09:46:09][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 09:46:09][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 09:46:09][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step378 to Phase-Technologies/forge-500m-pretrain (path: phase2_step378)... [2026-10-20 09:46:09][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 09:46:19][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 09:46:19][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step378 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 09:46:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5298 [2026-10-20 09:46:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6711 [2026-10-20 09:46:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5761 [2026-10-20 09:46:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4634 [2026-10-20 09:46:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4013 [2026-10-20 09:46:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3117 [2026-10-20 09:46:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5485 [2026-10-20 09:46:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2545 [2026-10-20 09:46:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7901 [2026-10-20 09:46:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6385 [2026-10-20 09:46:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8476 [2026-10-20 09:47:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6991 [2026-10-20 09:47:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5333 [2026-10-20 09:47:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4488 [2026-10-20 09:47:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0546 [2026-10-20 09:47:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9472 [2026-10-20 09:47:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5508 [2026-10-20 09:47:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2099 [2026-10-20 09:47:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1032 [2026-10-20 09:47:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0701 [2026-10-20 09:47:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0313 [2026-10-20 09:47:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0090 [2026-10-20 09:47:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8140 [2026-10-20 09:47:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4038 [2026-10-20 09:48:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1068 [2026-10-20 09:48:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4310 [2026-10-20 09:48:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6184 [2026-10-20 09:48:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0367 [2026-10-20 09:48:28][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,497 | MFU: 1.8% [2026-10-20 09:48:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8734 [2026-10-20 09:48:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5616 [2026-10-20 09:48:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1122 [2026-10-20 09:48:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0694 [2026-10-20 09:48:33][INFO][training.pretrain_engine] [Ph2|Step 379] loss=6.2442 | ppl=515.01 | aux=0.02514 | lr=2.49e-04 | gnorm=1.289 | tok/s=29,198 | MFU=7.0% | step=143649ms | tokens=0.405B | mem=4.2/32.7GB | cost=$199.09/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 04:39 (138.9h) | Phase3: SKIPPED | SFT: Oct 26 09:25 (143.6h) | DPO (Final): Oct 26 09:59 (144.2h) [2026-10-20 09:48:33][INFO][training.pretrain_engine] Starting global step 380 (accumulating 32 micro-batches)... [2026-10-20 09:48:33][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 09:48:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3195 [2026-10-20 09:48:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1967 [2026-10-20 09:48:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9719 [2026-10-20 09:48:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3175 [2026-10-20 09:49:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5785 [2026-10-20 09:49:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3833 [2026-10-20 09:49:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1560 [2026-10-20 09:49:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4342 [2026-10-20 09:49:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3220 [2026-10-20 09:49:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5071 [2026-10-20 09:49:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0485 [2026-10-20 09:49:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4834 [2026-10-20 09:49:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1899 [2026-10-20 09:49:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5884 [2026-10-20 09:49:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1672 [2026-10-20 09:49:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1995 [2026-10-20 09:50:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5752 [2026-10-20 09:50:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5893 [2026-10-20 09:50:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4570 [2026-10-20 09:50:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5616 [2026-10-20 09:50:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8789 [2026-10-20 09:50:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2085 [2026-10-20 09:50:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0717 [2026-10-20 09:50:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8819 [2026-10-20 09:50:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8722 [2026-10-20 09:50:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4587 [2026-10-20 09:50:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0634 [2026-10-20 09:50:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1029 [2026-10-20 09:50:53][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,511 | MFU: 1.8% [2026-10-20 09:50:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6112 [2026-10-20 09:50:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5753 [2026-10-20 09:50:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4093 [2026-10-20 09:50:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7648 [2026-10-20 09:50:57][INFO][training.pretrain_engine] [Ph2|Step 380] loss=6.2808 | ppl=534.24 | aux=0.02513 | lr=2.49e-04 | gnorm=1.430 | tok/s=29,196 | MFU=7.0% | step=143659ms | tokens=0.406B | mem=4.2/32.7GB | cost=$199.62/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 04:42 (138.9h) | Phase3: SKIPPED | SFT: Oct 26 09:27 (143.6h) | DPO (Final): Oct 26 10:01 (144.2h) [2026-10-20 09:50:57][INFO][training.pretrain_engine] Starting global step 381 (accumulating 32 micro-batches)... [2026-10-20 09:50:57][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 09:51:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4197 [2026-10-20 09:51:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0447 [2026-10-20 09:51:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4593 [2026-10-20 09:51:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8458 [2026-10-20 09:51:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8205 [2026-10-20 09:51:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1449 [2026-10-20 09:51:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5625 [2026-10-20 09:51:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0742 [2026-10-20 09:51:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3509 [2026-10-20 09:51:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1430 [2026-10-20 09:51:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3248 [2026-10-20 09:51:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8293 [2026-10-20 09:52:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2326 [2026-10-20 09:52:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9482 [2026-10-20 09:52:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9931 [2026-10-20 09:52:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4877 [2026-10-20 09:52:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5526 [2026-10-20 09:52:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2635 [2026-10-20 09:52:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5556 [2026-10-20 09:52:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4942 [2026-10-20 09:52:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4295 [2026-10-20 09:52:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3536 [2026-10-20 09:52:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5711 [2026-10-20 09:52:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2999 [2026-10-20 09:53:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1382 [2026-10-20 09:53:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0078 [2026-10-20 09:53:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3584 [2026-10-20 09:53:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2588 [2026-10-20 09:53:16][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,528 | MFU: 1.8% [2026-10-20 09:53:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1877 [2026-10-20 09:53:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3803 [2026-10-20 09:53:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9771 [2026-10-20 09:53:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1000 [2026-10-20 09:53:21][INFO][training.pretrain_engine] [Ph2|Step 381] loss=6.1751 | ppl=480.62 | aux=0.02511 | lr=2.50e-04 | gnorm=0.953 | tok/s=29,195 | MFU=7.0% | step=143664ms | tokens=0.407B | mem=4.2/32.7GB | cost=$200.15/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 04:44 (138.8h) | Phase3: SKIPPED | SFT: Oct 26 09:29 (143.6h) | DPO (Final): Oct 26 10:03 (144.2h) [2026-10-20 09:53:21][INFO][training.pretrain_engine] Starting global step 382 (accumulating 32 micro-batches)... [2026-10-20 09:53:21][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 09:53:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3138 [2026-10-20 09:53:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3352 [2026-10-20 09:53:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5969 [2026-10-20 09:53:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7126 [2026-10-20 09:53:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1225 [2026-10-20 09:53:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2383 [2026-10-20 09:53:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1492 [2026-10-20 09:53:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4842 [2026-10-20 09:54:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5422 [2026-10-20 09:54:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4752 [2026-10-20 09:54:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7128 [2026-10-20 09:54:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4398 [2026-10-20 09:54:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1837 [2026-10-20 09:54:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7794 [2026-10-20 09:54:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1815 [2026-10-20 09:54:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9280 [2026-10-20 09:54:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1926 [2026-10-20 09:54:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1495 [2026-10-20 09:54:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1864 [2026-10-20 09:54:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0323 [2026-10-20 09:55:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8571 [2026-10-20 09:55:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1470 [2026-10-20 09:55:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7684 [2026-10-20 09:55:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4346 [2026-10-20 09:55:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5794 [2026-10-20 09:55:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0052 [2026-10-20 09:55:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3979 [2026-10-20 09:55:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5342 [2026-10-20 09:55:40][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,524 | MFU: 1.8% [2026-10-20 09:55:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3657 [2026-10-20 09:55:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6111 [2026-10-20 09:55:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3778 [2026-10-20 09:55:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1991 [2026-10-20 09:55:45][INFO][training.pretrain_engine] [Ph2|Step 382] loss=6.3784 | ppl=589.01 | aux=0.02514 | lr=2.50e-04 | gnorm=1.320 | tok/s=29,194 | MFU=7.0% | step=143672ms | tokens=0.408B | mem=4.2/32.7GB | cost=$200.67/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 04:46 (138.8h) | Phase3: SKIPPED | SFT: Oct 26 09:31 (143.6h) | DPO (Final): Oct 26 10:06 (144.2h) [2026-10-20 09:55:45][INFO][training.pretrain_engine] Starting global step 383 (accumulating 32 micro-batches)... [2026-10-20 09:55:45][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 09:56:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1452 [2026-10-20 09:56:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4153 [2026-10-20 09:56:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9910 [2026-10-20 09:56:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2257 [2026-10-20 09:56:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4528 [2026-10-20 09:56:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2855 [2026-10-20 09:56:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5758 [2026-10-20 09:56:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1316 [2026-10-20 09:56:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3617 [2026-10-20 09:56:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4143 [2026-10-20 09:56:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2703 [2026-10-20 09:56:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6462 [2026-10-20 09:56:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1329 [2026-10-20 09:56:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3691 [2026-10-20 09:56:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1078 [2026-10-20 09:56:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3360 [2026-10-20 09:57:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1354 [2026-10-20 09:57:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3350 [2026-10-20 09:57:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4069 [2026-10-20 09:57:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9988 [2026-10-20 09:57:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5163 [2026-10-20 09:57:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8855 [2026-10-20 09:57:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9359 [2026-10-20 09:57:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4772 [2026-10-20 09:57:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2242 [2026-10-20 09:57:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3971 [2026-10-20 09:57:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9750 [2026-10-20 09:57:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0079 [2026-10-20 09:58:04][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,532 | MFU: 1.8% [2026-10-20 09:58:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7660 [2026-10-20 09:58:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7179 [2026-10-20 09:58:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5367 [2026-10-20 09:58:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.9885 [2026-10-20 09:58:09][INFO][training.pretrain_engine] [Ph2|Step 383] loss=6.4137 | ppl=610.14 | aux=0.02512 | lr=2.51e-04 | gnorm=1.016 | tok/s=29,193 | MFU=7.0% | step=143677ms | tokens=0.409B | mem=4.2/32.7GB | cost=$201.20/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 04:48 (138.8h) | Phase3: SKIPPED | SFT: Oct 26 09:33 (143.6h) | DPO (Final): Oct 26 10:08 (144.2h) [2026-10-20 09:58:09][INFO][training.pretrain_engine] Starting global step 384 (accumulating 32 micro-batches)... [2026-10-20 09:58:09][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 09:58:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9019 [2026-10-20 09:58:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6783 [2026-10-20 09:58:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6718 [2026-10-20 09:58:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8067 [2026-10-20 09:58:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3288 [2026-10-20 09:58:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5200 [2026-10-20 09:58:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7537 [2026-10-20 09:58:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1860 [2026-10-20 09:59:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8822 [2026-10-20 09:59:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6649 [2026-10-20 09:59:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2839 [2026-10-20 09:59:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6634 [2026-10-20 09:59:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2082 [2026-10-20 09:59:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7823 [2026-10-20 09:59:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2872 [2026-10-20 09:59:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3141 [2026-10-20 09:59:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3289 [2026-10-20 09:59:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3412 [2026-10-20 09:59:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3543 [2026-10-20 09:59:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0868 [2026-10-20 09:59:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1761 [2026-10-20 09:59:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0330 [2026-10-20 09:59:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3522 [2026-10-20 09:59:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9668 [2026-10-20 10:00:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1878 [2026-10-20 10:00:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2220 [2026-10-20 10:00:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3254 [2026-10-20 10:00:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2520 [2026-10-20 10:00:28][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,518 | MFU: 1.8% [2026-10-20 10:00:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2989 [2026-10-20 10:00:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6098 [2026-10-20 10:00:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7225 [2026-10-20 10:00:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5459 [2026-10-20 10:00:33][INFO][training.pretrain_engine] [Ph2|Step 384] loss=6.2573 | ppl=521.82 | aux=0.02512 | lr=2.51e-04 | gnorm=1.078 | tok/s=29,191 | MFU=7.0% | step=143684ms | tokens=0.410B | mem=4.2/32.7GB | cost=$201.72/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 04:50 (138.8h) | Phase3: SKIPPED | SFT: Oct 26 09:36 (143.6h) | DPO (Final): Oct 26 10:10 (144.2h) [2026-10-20 10:00:33][INFO][training.pretrain_engine] Starting global step 385 (accumulating 32 micro-batches)... [2026-10-20 10:00:33][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 10:00:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2971 [2026-10-20 10:00:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8715 [2026-10-20 10:00:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3646 [2026-10-20 10:00:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1727 [2026-10-20 10:01:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2261 [2026-10-20 10:01:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8009 [2026-10-20 10:01:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9718 [2026-10-20 10:01:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7248 [2026-10-20 10:01:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1664 [2026-10-20 10:01:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5462 [2026-10-20 10:01:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3844 [2026-10-20 10:01:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1459 [2026-10-20 10:01:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9736 [2026-10-20 10:01:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6442 [2026-10-20 10:01:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8899 [2026-10-20 10:01:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3084 [2026-10-20 10:02:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4109 [2026-10-20 10:02:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3789 [2026-10-20 10:02:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4312 [2026-10-20 10:02:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6476 [2026-10-20 10:02:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5897 [2026-10-20 10:02:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1652 [2026-10-20 10:02:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6817 [2026-10-20 10:02:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8266 [2026-10-20 10:02:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0696 [2026-10-20 10:02:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6640 [2026-10-20 10:02:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1302 [2026-10-20 10:02:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6967 [2026-10-20 10:02:52][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,526 | MFU: 1.8% [2026-10-20 10:02:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7273 [2026-10-20 10:02:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4442 [2026-10-20 10:02:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8874 [2026-10-20 10:02:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4008 [2026-10-20 10:02:56][INFO][training.pretrain_engine] [Ph2|Step 385] loss=6.2311 | ppl=508.33 | aux=0.02513 | lr=2.52e-04 | gnorm=1.414 | tok/s=29,189 | MFU=7.0% | step=143692ms | tokens=0.411B | mem=4.2/32.7GB | cost=$202.25/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 04:52 (138.8h) | Phase3: SKIPPED | SFT: Oct 26 09:38 (143.6h) | DPO (Final): Oct 26 10:12 (144.2h) [2026-10-20 10:02:56][INFO][training.pretrain_engine] Starting global step 386 (accumulating 32 micro-batches)... [2026-10-20 10:02:56][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 10:03:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1762 [2026-10-20 10:03:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6048 [2026-10-20 10:03:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1191 [2026-10-20 10:03:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2952 [2026-10-20 10:03:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1556 [2026-10-20 10:03:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7126 [2026-10-20 10:03:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5776 [2026-10-20 10:03:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9867 [2026-10-20 10:03:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5874 [2026-10-20 10:03:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6261 [2026-10-20 10:03:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1864 [2026-10-20 10:03:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0854 [2026-10-20 10:04:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8769 [2026-10-20 10:04:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6908 [2026-10-20 10:04:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9122 [2026-10-20 10:04:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1292 [2026-10-20 10:04:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2777 [2026-10-20 10:04:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2982 [2026-10-20 10:04:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3870 [2026-10-20 10:04:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4503 [2026-10-20 10:04:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7802 [2026-10-20 10:04:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6586 [2026-10-20 10:04:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5133 [2026-10-20 10:04:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5069 [2026-10-20 10:05:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8697 [2026-10-20 10:05:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7741 [2026-10-20 10:05:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7356 [2026-10-20 10:05:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2846 [2026-10-20 10:05:16][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,525 | MFU: 1.8% [2026-10-20 10:05:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3837 [2026-10-20 10:05:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0897 [2026-10-20 10:05:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9721 [2026-10-20 10:05:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6768 [2026-10-20 10:05:21][INFO][training.pretrain_engine] [Ph2|Step 386] loss=6.2188 | ppl=502.09 | aux=0.02515 | lr=2.53e-04 | gnorm=1.141 | tok/s=29,189 | MFU=7.0% | step=143697ms | tokens=0.412B | mem=4.2/32.7GB | cost=$202.78/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 04:55 (138.8h) | Phase3: SKIPPED | SFT: Oct 26 09:40 (143.6h) | DPO (Final): Oct 26 10:14 (144.2h) [2026-10-20 10:05:21][INFO][training.pretrain_engine] Starting global step 387 (accumulating 32 micro-batches)... [2026-10-20 10:05:21][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 10:05:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5786 [2026-10-20 10:05:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1095 [2026-10-20 10:05:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1606 [2026-10-20 10:05:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7091 [2026-10-20 10:05:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3063 [2026-10-20 10:05:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9054 [2026-10-20 10:05:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7047 [2026-10-20 10:05:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5945 [2026-10-20 10:06:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3923 [2026-10-20 10:06:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1800 [2026-10-20 10:06:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1183 [2026-10-20 10:06:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1078 [2026-10-20 10:06:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9813 [2026-10-20 10:06:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1094 [2026-10-20 10:06:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1470 [2026-10-20 10:06:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7532 [2026-10-20 10:06:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3244 [2026-10-20 10:06:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2279 [2026-10-20 10:06:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5236 [2026-10-20 10:06:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2948 [2026-10-20 10:07:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7620 [2026-10-20 10:07:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6705 [2026-10-20 10:07:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5758 [2026-10-20 10:07:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1642 [2026-10-20 10:07:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9402 [2026-10-20 10:07:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2443 [2026-10-20 10:07:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6874 [2026-10-20 10:07:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0065 [2026-10-20 10:07:40][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,528 | MFU: 1.8% [2026-10-20 10:07:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5567 [2026-10-20 10:07:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9570 [2026-10-20 10:07:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7926 [2026-10-20 10:07:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1547 [2026-10-20 10:07:44][INFO][training.pretrain_engine] [Ph2|Step 387] loss=6.2539 | ppl=520.02 | aux=0.02515 | lr=2.53e-04 | gnorm=1.094 | tok/s=29,187 | MFU=7.0% | step=143703ms | tokens=0.413B | mem=4.2/32.7GB | cost=$203.30/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 04:57 (138.8h) | Phase3: SKIPPED | SFT: Oct 26 09:42 (143.6h) | DPO (Final): Oct 26 10:16 (144.2h) [2026-10-20 10:07:48][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step387 | Tokens: 0.413B | Spent: $203.31 | Remaining: $196.69 [2026-10-20 10:07:48][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step387 [2026-10-20 10:07:48][INFO][training.pretrain_engine] Starting global step 388 (accumulating 32 micro-batches)... [2026-10-20 10:07:48][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 10:07:48][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 10:07:48][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 10:07:48][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 10:07:48][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step387 to Phase-Technologies/forge-500m-pretrain (path: phase2_step387)... [2026-10-20 10:07:48][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 10:07:58][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 10:07:58][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step387 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 10:08:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1733 [2026-10-20 10:08:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2881 [2026-10-20 10:08:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1052 [2026-10-20 10:08:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9618 [2026-10-20 10:08:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3064 [2026-10-20 10:08:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2809 [2026-10-20 10:08:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9190 [2026-10-20 10:08:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2756 [2026-10-20 10:08:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5102 [2026-10-20 10:08:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8287 [2026-10-20 10:08:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4184 [2026-10-20 10:08:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8948 [2026-10-20 10:08:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3586 [2026-10-20 10:08:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2506 [2026-10-20 10:08:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3498 [2026-10-20 10:08:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4068 [2026-10-20 10:09:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0522 [2026-10-20 10:09:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1514 [2026-10-20 10:09:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5204 [2026-10-20 10:09:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0343 [2026-10-20 10:09:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3444 [2026-10-20 10:09:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9957 [2026-10-20 10:09:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3282 [2026-10-20 10:09:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4755 [2026-10-20 10:09:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1613 [2026-10-20 10:09:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2452 [2026-10-20 10:09:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3358 [2026-10-20 10:09:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2697 [2026-10-20 10:10:07][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,545 | MFU: 1.8% [2026-10-20 10:10:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8174 [2026-10-20 10:10:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0132 [2026-10-20 10:10:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3346 [2026-10-20 10:10:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3625 [2026-10-20 10:10:11][INFO][training.pretrain_engine] [Ph2|Step 388] loss=6.3227 | ppl=557.07 | aux=0.02515 | lr=2.54e-04 | gnorm=0.938 | tok/s=29,187 | MFU=7.0% | step=143705ms | tokens=0.414B | mem=4.2/32.7GB | cost=$203.84/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 04:59 (138.8h) | Phase3: SKIPPED | SFT: Oct 26 09:44 (143.6h) | DPO (Final): Oct 26 10:18 (144.1h) [2026-10-20 10:10:11][INFO][training.pretrain_engine] Starting global step 389 (accumulating 32 micro-batches)... [2026-10-20 10:10:11][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 10:10:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9723 [2026-10-20 10:10:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1718 [2026-10-20 10:10:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9390 [2026-10-20 10:10:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1584 [2026-10-20 10:10:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2351 [2026-10-20 10:10:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2866 [2026-10-20 10:10:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2703 [2026-10-20 10:10:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2384 [2026-10-20 10:11:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2859 [2026-10-20 10:11:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2807 [2026-10-20 10:11:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3730 [2026-10-20 10:11:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4838 [2026-10-20 10:11:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1404 [2026-10-20 10:11:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2803 [2026-10-20 10:11:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9219 [2026-10-20 10:11:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2088 [2026-10-20 10:11:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2299 [2026-10-20 10:11:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9266 [2026-10-20 10:11:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1273 [2026-10-20 10:11:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3661 [2026-10-20 10:11:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6701 [2026-10-20 10:11:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9803 [2026-10-20 10:11:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8124 [2026-10-20 10:11:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1498 [2026-10-20 10:12:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1605 [2026-10-20 10:12:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2905 [2026-10-20 10:12:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7222 [2026-10-20 10:12:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3252 [2026-10-20 10:12:30][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,556 | MFU: 1.8% [2026-10-20 10:12:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1373 [2026-10-20 10:12:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1311 [2026-10-20 10:12:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3860 [2026-10-20 10:12:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0745 [2026-10-20 10:12:35][INFO][training.pretrain_engine] [Ph2|Step 389] loss=6.2471 | ppl=516.49 | aux=0.02512 | lr=2.54e-04 | gnorm=1.430 | tok/s=29,186 | MFU=7.0% | step=143709ms | tokens=0.415B | mem=4.2/32.7GB | cost=$204.36/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 05:01 (138.8h) | Phase3: SKIPPED | SFT: Oct 26 09:46 (143.6h) | DPO (Final): Oct 26 10:20 (144.1h) [2026-10-20 10:12:35][INFO][training.pretrain_engine] Starting global step 390 (accumulating 32 micro-batches)... [2026-10-20 10:12:35][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 10:12:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3228 [2026-10-20 10:12:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8189 [2026-10-20 10:12:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2816 [2026-10-20 10:12:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3104 [2026-10-20 10:13:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7096 [2026-10-20 10:13:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0854 [2026-10-20 10:13:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6036 [2026-10-20 10:13:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6461 [2026-10-20 10:13:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1381 [2026-10-20 10:13:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0811 [2026-10-20 10:13:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4371 [2026-10-20 10:13:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1658 [2026-10-20 10:13:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6521 [2026-10-20 10:13:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5698 [2026-10-20 10:13:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5813 [2026-10-20 10:13:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7111 [2026-10-20 10:14:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4583 [2026-10-20 10:14:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2267 [2026-10-20 10:14:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4727 [2026-10-20 10:14:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2189 [2026-10-20 10:14:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1547 [2026-10-20 10:14:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0648 [2026-10-20 10:14:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4540 [2026-10-20 10:14:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8678 [2026-10-20 10:14:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7620 [2026-10-20 10:14:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6518 [2026-10-20 10:14:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1506 [2026-10-20 10:14:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5685 [2026-10-20 10:14:53][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,577 | MFU: 1.8% [2026-10-20 10:14:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7175 [2026-10-20 10:14:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2062 [2026-10-20 10:14:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0319 [2026-10-20 10:14:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7877 [2026-10-20 10:14:58][INFO][training.pretrain_engine] [Ph2|Step 390] loss=6.3451 | ppl=569.72 | aux=0.02512 | lr=2.55e-04 | gnorm=0.766 | tok/s=29,186 | MFU=7.0% | step=143711ms | tokens=0.416B | mem=4.2/32.7GB | cost=$204.89/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 05:03 (138.8h) | Phase3: SKIPPED | SFT: Oct 26 09:48 (143.6h) | DPO (Final): Oct 26 10:22 (144.1h) [2026-10-20 10:14:58][INFO][training.pretrain_engine] Starting global step 391 (accumulating 32 micro-batches)... [2026-10-20 10:14:58][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 10:15:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0781 [2026-10-20 10:15:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8586 [2026-10-20 10:15:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1740 [2026-10-20 10:15:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2124 [2026-10-20 10:15:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3820 [2026-10-20 10:15:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9657 [2026-10-20 10:15:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8228 [2026-10-20 10:15:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9320 [2026-10-20 10:15:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6963 [2026-10-20 10:15:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5408 [2026-10-20 10:15:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4388 [2026-10-20 10:15:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1890 [2026-10-20 10:16:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8166 [2026-10-20 10:16:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7028 [2026-10-20 10:16:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3673 [2026-10-20 10:16:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3527 [2026-10-20 10:16:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2391 [2026-10-20 10:16:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0577 [2026-10-20 10:16:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0303 [2026-10-20 10:16:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3873 [2026-10-20 10:16:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8868 [2026-10-20 10:16:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2554 [2026-10-20 10:16:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2842 [2026-10-20 10:16:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0743 [2026-10-20 10:17:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4085 [2026-10-20 10:17:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2602 [2026-10-20 10:17:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2032 [2026-10-20 10:17:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4204 [2026-10-20 10:17:17][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,565 | MFU: 1.8% [2026-10-20 10:17:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7462 [2026-10-20 10:17:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7440 [2026-10-20 10:17:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3178 [2026-10-20 10:17:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2121 [2026-10-20 10:17:22][INFO][training.pretrain_engine] [Ph2|Step 391] loss=6.2651 | ppl=525.90 | aux=0.02512 | lr=2.55e-04 | gnorm=1.555 | tok/s=29,186 | MFU=7.0% | step=143711ms | tokens=0.417B | mem=4.2/32.7GB | cost=$205.41/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 05:04 (138.8h) | Phase3: SKIPPED | SFT: Oct 26 09:50 (143.6h) | DPO (Final): Oct 26 10:24 (144.1h) [2026-10-20 10:17:22][INFO][training.pretrain_engine] Starting global step 392 (accumulating 32 micro-batches)... [2026-10-20 10:17:22][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 10:17:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3257 [2026-10-20 10:17:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5505 [2026-10-20 10:17:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2845 [2026-10-20 10:17:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2482 [2026-10-20 10:17:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8116 [2026-10-20 10:17:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7894 [2026-10-20 10:17:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5510 [2026-10-20 10:17:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9892 [2026-10-20 10:18:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1984 [2026-10-20 10:18:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3717 [2026-10-20 10:18:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5313 [2026-10-20 10:18:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4463 [2026-10-20 10:18:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3843 [2026-10-20 10:18:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7196 [2026-10-20 10:18:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5299 [2026-10-20 10:18:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5083 [2026-10-20 10:18:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2446 [2026-10-20 10:18:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2165 [2026-10-20 10:18:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5703 [2026-10-20 10:18:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4682 [2026-10-20 10:19:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7324 [2026-10-20 10:19:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8287 [2026-10-20 10:19:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5223 [2026-10-20 10:19:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0348 [2026-10-20 10:19:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4048 [2026-10-20 10:19:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1896 [2026-10-20 10:19:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7386 [2026-10-20 10:19:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0239 [2026-10-20 10:19:41][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,521 | MFU: 1.8% [2026-10-20 10:19:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1899 [2026-10-20 10:19:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3609 [2026-10-20 10:19:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6335 [2026-10-20 10:19:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6180 [2026-10-20 10:19:46][INFO][training.pretrain_engine] [Ph2|Step 392] loss=6.3215 | ppl=556.43 | aux=0.02514 | lr=2.56e-04 | gnorm=0.902 | tok/s=29,185 | MFU=7.0% | step=143717ms | tokens=0.418B | mem=4.2/32.7GB | cost=$205.94/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 05:07 (138.8h) | Phase3: SKIPPED | SFT: Oct 26 09:52 (143.5h) | DPO (Final): Oct 26 10:26 (144.1h) [2026-10-20 10:19:46][INFO][training.pretrain_engine] Starting global step 393 (accumulating 32 micro-batches)... [2026-10-20 10:19:46][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 10:20:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7713 [2026-10-20 10:20:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1087 [2026-10-20 10:20:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8541 [2026-10-20 10:20:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6782 [2026-10-20 10:20:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1538 [2026-10-20 10:20:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3582 [2026-10-20 10:20:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0260 [2026-10-20 10:20:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0222 [2026-10-20 10:20:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4237 [2026-10-20 10:20:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8646 [2026-10-20 10:20:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2160 [2026-10-20 10:20:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4156 [2026-10-20 10:20:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4958 [2026-10-20 10:20:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0723 [2026-10-20 10:20:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7322 [2026-10-20 10:20:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9369 [2026-10-20 10:21:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1850 [2026-10-20 10:21:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1257 [2026-10-20 10:21:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9611 [2026-10-20 10:21:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2581 [2026-10-20 10:21:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3824 [2026-10-20 10:21:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2569 [2026-10-20 10:21:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8815 [2026-10-20 10:21:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5380 [2026-10-20 10:21:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1666 [2026-10-20 10:21:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2851 [2026-10-20 10:21:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6057 [2026-10-20 10:21:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0668 [2026-10-20 10:22:05][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,530 | MFU: 1.8% [2026-10-20 10:22:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9379 [2026-10-20 10:22:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3010 [2026-10-20 10:22:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2125 [2026-10-20 10:22:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7291 [2026-10-20 10:22:10][INFO][training.pretrain_engine] [Ph2|Step 393] loss=6.2116 | ppl=498.49 | aux=0.02515 | lr=2.56e-04 | gnorm=1.305 | tok/s=29,183 | MFU=7.0% | step=143724ms | tokens=0.419B | mem=4.2/32.7GB | cost=$206.47/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 05:09 (138.8h) | Phase3: SKIPPED | SFT: Oct 26 09:54 (143.5h) | DPO (Final): Oct 26 10:29 (144.1h) [2026-10-20 10:22:10][INFO][training.pretrain_engine] Starting global step 394 (accumulating 32 micro-batches)... [2026-10-20 10:22:10][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 10:22:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6682 [2026-10-20 10:22:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5353 [2026-10-20 10:22:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.3016 [2026-10-20 10:22:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5640 [2026-10-20 10:22:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3714 [2026-10-20 10:22:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2438 [2026-10-20 10:22:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2118 [2026-10-20 10:22:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5283 [2026-10-20 10:23:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1431 [2026-10-20 10:23:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2634 [2026-10-20 10:23:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5796 [2026-10-20 10:23:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4348 [2026-10-20 10:23:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4277 [2026-10-20 10:23:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7070 [2026-10-20 10:23:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5519 [2026-10-20 10:23:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4442 [2026-10-20 10:23:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0492 [2026-10-20 10:23:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2267 [2026-10-20 10:23:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1858 [2026-10-20 10:23:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9416 [2026-10-20 10:23:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4926 [2026-10-20 10:23:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2309 [2026-10-20 10:23:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2866 [2026-10-20 10:23:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3583 [2026-10-20 10:24:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4669 [2026-10-20 10:24:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2788 [2026-10-20 10:24:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0815 [2026-10-20 10:24:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2678 [2026-10-20 10:24:28][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,582 | MFU: 1.8% [2026-10-20 10:24:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0606 [2026-10-20 10:24:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5569 [2026-10-20 10:24:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2653 [2026-10-20 10:24:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3195 [2026-10-20 10:24:33][INFO][training.pretrain_engine] [Ph2|Step 394] loss=6.3668 | ppl=582.21 | aux=0.02515 | lr=2.57e-04 | gnorm=1.062 | tok/s=29,183 | MFU=7.0% | step=143725ms | tokens=0.420B | mem=4.2/32.7GB | cost=$206.99/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 05:11 (138.8h) | Phase3: SKIPPED | SFT: Oct 26 09:56 (143.5h) | DPO (Final): Oct 26 10:30 (144.1h) [2026-10-20 10:24:33][INFO][training.pretrain_engine] Starting global step 395 (accumulating 32 micro-batches)... [2026-10-20 10:24:33][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 10:24:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0979 [2026-10-20 10:24:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9179 [2026-10-20 10:24:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6816 [2026-10-20 10:24:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5282 [2026-10-20 10:25:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0777 [2026-10-20 10:25:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5333 [2026-10-20 10:25:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4847 [2026-10-20 10:25:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.1327 [2026-10-20 10:25:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1178 [2026-10-20 10:25:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2640 [2026-10-20 10:25:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2576 [2026-10-20 10:25:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8948 [2026-10-20 10:25:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2564 [2026-10-20 10:25:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7958 [2026-10-20 10:25:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1431 [2026-10-20 10:25:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6974 [2026-10-20 10:26:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8436 [2026-10-20 10:26:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7127 [2026-10-20 10:26:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2961 [2026-10-20 10:26:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0290 [2026-10-20 10:26:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2746 [2026-10-20 10:26:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0463 [2026-10-20 10:26:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1305 [2026-10-20 10:26:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1736 [2026-10-20 10:26:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7309 [2026-10-20 10:26:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1967 [2026-10-20 10:26:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4061 [2026-10-20 10:26:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6582 [2026-10-20 10:26:51][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,580 | MFU: 1.8% [2026-10-20 10:26:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3718 [2026-10-20 10:26:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3110 [2026-10-20 10:26:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9611 [2026-10-20 10:26:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0549 [2026-10-20 10:26:57][INFO][training.pretrain_engine] [Ph2|Step 395] loss=6.3132 | ppl=551.81 | aux=0.02512 | lr=2.58e-04 | gnorm=0.934 | tok/s=29,182 | MFU=7.0% | step=143727ms | tokens=0.422B | mem=4.2/32.7GB | cost=$207.52/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 05:13 (138.8h) | Phase3: SKIPPED | SFT: Oct 26 09:58 (143.5h) | DPO (Final): Oct 26 10:32 (144.1h) [2026-10-20 10:26:57][INFO][training.pretrain_engine] Starting global step 396 (accumulating 32 micro-batches)... [2026-10-20 10:26:57][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 10:27:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7451 [2026-10-20 10:27:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5062 [2026-10-20 10:27:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5938 [2026-10-20 10:27:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6804 [2026-10-20 10:27:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3949 [2026-10-20 10:27:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4404 [2026-10-20 10:27:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0972 [2026-10-20 10:27:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5996 [2026-10-20 10:27:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7062 [2026-10-20 10:27:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0997 [2026-10-20 10:27:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0978 [2026-10-20 10:27:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7542 [2026-10-20 10:28:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3385 [2026-10-20 10:28:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7519 [2026-10-20 10:28:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.2414 [2026-10-20 10:28:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4222 [2026-10-20 10:28:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0075 [2026-10-20 10:28:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5877 [2026-10-20 10:28:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1369 [2026-10-20 10:28:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8850 [2026-10-20 10:28:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2902 [2026-10-20 10:28:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3150 [2026-10-20 10:28:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6814 [2026-10-20 10:28:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0359 [2026-10-20 10:29:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2372 [2026-10-20 10:29:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6988 [2026-10-20 10:29:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8032 [2026-10-20 10:29:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7506 [2026-10-20 10:29:15][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,586 | MFU: 1.8% [2026-10-20 10:29:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9921 [2026-10-20 10:29:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2566 [2026-10-20 10:29:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5416 [2026-10-20 10:29:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2218 [2026-10-20 10:29:20][INFO][training.pretrain_engine] [Ph2|Step 396] loss=6.3626 | ppl=579.78 | aux=0.02513 | lr=2.58e-04 | gnorm=1.109 | tok/s=29,182 | MFU=7.0% | step=143729ms | tokens=0.423B | mem=4.2/32.7GB | cost=$208.04/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 05:14 (138.8h) | Phase3: SKIPPED | SFT: Oct 26 10:00 (143.5h) | DPO (Final): Oct 26 10:34 (144.1h) [2026-10-20 10:29:24][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step396 | Tokens: 0.423B | Spent: $208.05 | Remaining: $191.95 [2026-10-20 10:29:24][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step396 [2026-10-20 10:29:24][INFO][training.pretrain_engine] Starting global step 397 (accumulating 32 micro-batches)... [2026-10-20 10:29:24][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 10:29:24][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 10:29:24][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 10:29:24][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 10:29:24][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step396 to Phase-Technologies/forge-500m-pretrain (path: phase2_step396)... [2026-10-20 10:29:24][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 10:29:34][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 10:29:34][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step396 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 10:29:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9473 [2026-10-20 10:29:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3298 [2026-10-20 10:29:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2740 [2026-10-20 10:29:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5892 [2026-10-20 10:29:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6513 [2026-10-20 10:29:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6686 [2026-10-20 10:29:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7048 [2026-10-20 10:30:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.0510 [2026-10-20 10:30:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7918 [2026-10-20 10:30:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8569 [2026-10-20 10:30:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6635 [2026-10-20 10:30:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1165 [2026-10-20 10:30:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7623 [2026-10-20 10:30:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0564 [2026-10-20 10:30:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1068 [2026-10-20 10:30:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7019 [2026-10-20 10:30:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9970 [2026-10-20 10:30:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4306 [2026-10-20 10:30:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5238 [2026-10-20 10:30:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5384 [2026-10-20 10:31:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8934 [2026-10-20 10:31:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0198 [2026-10-20 10:31:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6338 [2026-10-20 10:31:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4552 [2026-10-20 10:31:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2573 [2026-10-20 10:31:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9771 [2026-10-20 10:31:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3174 [2026-10-20 10:31:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3526 [2026-10-20 10:31:42][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,579 | MFU: 1.8% [2026-10-20 10:31:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9066 [2026-10-20 10:31:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7393 [2026-10-20 10:31:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7433 [2026-10-20 10:31:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9419 [2026-10-20 10:31:47][INFO][training.pretrain_engine] [Ph2|Step 397] loss=6.2165 | ppl=500.94 | aux=0.02514 | lr=2.59e-04 | gnorm=1.102 | tok/s=29,183 | MFU=7.0% | step=143726ms | tokens=0.424B | mem=4.2/32.7GB | cost=$208.58/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 05:16 (138.7h) | Phase3: SKIPPED | SFT: Oct 26 10:02 (143.5h) | DPO (Final): Oct 26 10:36 (144.1h) [2026-10-20 10:31:47][INFO][training.pretrain_engine] Starting global step 398 (accumulating 32 micro-batches)... [2026-10-20 10:31:47][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 10:32:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9676 [2026-10-20 10:32:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6160 [2026-10-20 10:32:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1230 [2026-10-20 10:32:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9323 [2026-10-20 10:32:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4365 [2026-10-20 10:32:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1979 [2026-10-20 10:32:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4170 [2026-10-20 10:32:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3080 [2026-10-20 10:32:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7062 [2026-10-20 10:32:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4013 [2026-10-20 10:32:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7182 [2026-10-20 10:32:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9711 [2026-10-20 10:32:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5119 [2026-10-20 10:32:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3537 [2026-10-20 10:32:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6443 [2026-10-20 10:32:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7882 [2026-10-20 10:33:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6621 [2026-10-20 10:33:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1580 [2026-10-20 10:33:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2736 [2026-10-20 10:33:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8579 [2026-10-20 10:33:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1197 [2026-10-20 10:33:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3910 [2026-10-20 10:33:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4152 [2026-10-20 10:33:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6390 [2026-10-20 10:33:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4862 [2026-10-20 10:33:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5695 [2026-10-20 10:33:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6538 [2026-10-20 10:33:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8146 [2026-10-20 10:34:06][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,531 | MFU: 1.8% [2026-10-20 10:34:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0822 [2026-10-20 10:34:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4099 [2026-10-20 10:34:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0100 [2026-10-20 10:34:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9421 [2026-10-20 10:34:11][INFO][training.pretrain_engine] [Ph2|Step 398] loss=6.2956 | ppl=542.19 | aux=0.02514 | lr=2.59e-04 | gnorm=0.898 | tok/s=29,181 | MFU=7.0% | step=143735ms | tokens=0.425B | mem=4.2/32.7GB | cost=$209.10/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 05:18 (138.7h) | Phase3: SKIPPED | SFT: Oct 26 10:04 (143.5h) | DPO (Final): Oct 26 10:38 (144.1h) [2026-10-20 10:34:11][INFO][training.pretrain_engine] Starting global step 399 (accumulating 32 micro-batches)... [2026-10-20 10:34:11][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 10:34:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3148 [2026-10-20 10:34:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6387 [2026-10-20 10:34:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2033 [2026-10-20 10:34:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4419 [2026-10-20 10:34:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8343 [2026-10-20 10:34:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6563 [2026-10-20 10:34:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7323 [2026-10-20 10:34:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5333 [2026-10-20 10:35:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4234 [2026-10-20 10:35:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7235 [2026-10-20 10:35:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.9772 [2026-10-20 10:35:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5643 [2026-10-20 10:35:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7420 [2026-10-20 10:35:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2721 [2026-10-20 10:35:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5813 [2026-10-20 10:35:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1525 [2026-10-20 10:35:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9826 [2026-10-20 10:35:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1442 [2026-10-20 10:35:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3459 [2026-10-20 10:35:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5437 [2026-10-20 10:35:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3286 [2026-10-20 10:35:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2410 [2026-10-20 10:35:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7849 [2026-10-20 10:35:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9779 [2026-10-20 10:36:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8894 [2026-10-20 10:36:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3056 [2026-10-20 10:36:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6264 [2026-10-20 10:36:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2532 [2026-10-20 10:36:30][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,534 | MFU: 1.8% [2026-10-20 10:36:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1974 [2026-10-20 10:36:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1687 [2026-10-20 10:36:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0731 [2026-10-20 10:36:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3409 [2026-10-20 10:36:34][INFO][training.pretrain_engine] [Ph2|Step 399] loss=6.2288 | ppl=507.13 | aux=0.02512 | lr=2.60e-04 | gnorm=0.941 | tok/s=29,180 | MFU=7.0% | step=143738ms | tokens=0.426B | mem=4.2/32.7GB | cost=$209.63/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 05:20 (138.7h) | Phase3: SKIPPED | SFT: Oct 26 10:06 (143.5h) | DPO (Final): Oct 26 10:40 (144.1h) [2026-10-20 10:36:34][INFO][training.pretrain_engine] Starting global step 400 (accumulating 32 micro-batches)... [2026-10-20 10:36:34][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 10:36:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8141 [2026-10-20 10:36:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3160 [2026-10-20 10:36:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4267 [2026-10-20 10:36:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5653 [2026-10-20 10:37:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9206 [2026-10-20 10:37:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8114 [2026-10-20 10:37:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1750 [2026-10-20 10:37:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5399 [2026-10-20 10:37:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3463 [2026-10-20 10:37:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7899 [2026-10-20 10:37:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0407 [2026-10-20 10:37:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2879 [2026-10-20 10:37:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5730 [2026-10-20 10:37:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8794 [2026-10-20 10:37:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3552 [2026-10-20 10:37:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9207 [2026-10-20 10:38:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5598 [2026-10-20 10:38:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1742 [2026-10-20 10:38:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1729 [2026-10-20 10:38:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2010 [2026-10-20 10:38:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2511 [2026-10-20 10:38:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4281 [2026-10-20 10:38:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3219 [2026-10-20 10:38:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2919 [2026-10-20 10:38:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.1298 [2026-10-20 10:38:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4137 [2026-10-20 10:38:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.4779 [2026-10-20 10:38:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3924 [2026-10-20 10:38:53][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,535 | MFU: 1.8% [2026-10-20 10:38:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1928 [2026-10-20 10:38:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0762 [2026-10-20 10:38:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0497 [2026-10-20 10:38:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9263 [2026-10-20 10:38:58][INFO][training.pretrain_engine] [Ph2|Step 400] loss=6.2060 | ppl=495.74 | aux=0.02510 | lr=2.60e-04 | gnorm=0.719 | tok/s=29,179 | MFU=7.0% | step=143743ms | tokens=0.427B | mem=4.2/32.7GB | cost=$210.15/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 05:22 (138.7h) | Phase3: SKIPPED | SFT: Oct 26 10:08 (143.5h) | DPO (Final): Oct 26 10:42 (144.1h) [2026-10-20 10:38:58][INFO][training.pretrain_engine] Starting global step 401 (accumulating 32 micro-batches)... [2026-10-20 10:38:58][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 10:39:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6680 [2026-10-20 10:39:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9118 [2026-10-20 10:39:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6850 [2026-10-20 10:39:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9121 [2026-10-20 10:39:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3023 [2026-10-20 10:39:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3535 [2026-10-20 10:39:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1382 [2026-10-20 10:39:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5941 [2026-10-20 10:39:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7119 [2026-10-20 10:39:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8241 [2026-10-20 10:39:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.5966 [2026-10-20 10:39:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4891 [2026-10-20 10:40:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5966 [2026-10-20 10:40:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4013 [2026-10-20 10:40:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3244 [2026-10-20 10:40:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4251 [2026-10-20 10:40:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2911 [2026-10-20 10:40:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0397 [2026-10-20 10:40:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8651 [2026-10-20 10:40:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1674 [2026-10-20 10:40:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3833 [2026-10-20 10:40:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4860 [2026-10-20 10:40:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6130 [2026-10-20 10:40:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6347 [2026-10-20 10:41:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7203 [2026-10-20 10:41:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5482 [2026-10-20 10:41:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5703 [2026-10-20 10:41:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0102 [2026-10-20 10:41:17][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,518 | MFU: 1.8% [2026-10-20 10:41:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7848 [2026-10-20 10:41:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0477 [2026-10-20 10:41:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2490 [2026-10-20 10:41:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6791 [2026-10-20 10:41:22][INFO][training.pretrain_engine] [Ph2|Step 401] loss=6.1694 | ppl=477.90 | aux=0.02516 | lr=2.61e-04 | gnorm=0.980 | tok/s=29,177 | MFU=7.0% | step=143754ms | tokens=0.428B | mem=4.2/32.7GB | cost=$210.68/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 05:25 (138.7h) | Phase3: SKIPPED | SFT: Oct 26 10:11 (143.5h) | DPO (Final): Oct 26 10:45 (144.1h) [2026-10-20 10:41:22][INFO][training.pretrain_engine] Starting global step 402 (accumulating 32 micro-batches)... [2026-10-20 10:41:22][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 10:41:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9756 [2026-10-20 10:41:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1314 [2026-10-20 10:41:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0861 [2026-10-20 10:41:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9996 [2026-10-20 10:41:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2406 [2026-10-20 10:41:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0958 [2026-10-20 10:41:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2374 [2026-10-20 10:41:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2481 [2026-10-20 10:42:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3934 [2026-10-20 10:42:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3152 [2026-10-20 10:42:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9097 [2026-10-20 10:42:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1808 [2026-10-20 10:42:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6210 [2026-10-20 10:42:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1481 [2026-10-20 10:42:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9830 [2026-10-20 10:42:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1121 [2026-10-20 10:42:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9608 [2026-10-20 10:42:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2605 [2026-10-20 10:42:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2152 [2026-10-20 10:42:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0163 [2026-10-20 10:43:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3134 [2026-10-20 10:43:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0742 [2026-10-20 10:43:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4295 [2026-10-20 10:43:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2918 [2026-10-20 10:43:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3863 [2026-10-20 10:43:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8668 [2026-10-20 10:43:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2801 [2026-10-20 10:43:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9376 [2026-10-20 10:43:41][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,565 | MFU: 1.8% [2026-10-20 10:43:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2376 [2026-10-20 10:43:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0687 [2026-10-20 10:43:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5045 [2026-10-20 10:43:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9418 [2026-10-20 10:43:45][INFO][training.pretrain_engine] [Ph2|Step 402] loss=6.1685 | ppl=477.48 | aux=0.02513 | lr=2.62e-04 | gnorm=1.156 | tok/s=29,177 | MFU=7.0% | step=143755ms | tokens=0.429B | mem=4.2/32.7GB | cost=$211.20/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 05:27 (138.7h) | Phase3: SKIPPED | SFT: Oct 26 10:12 (143.5h) | DPO (Final): Oct 26 10:47 (144.1h) [2026-10-20 10:43:45][INFO][training.pretrain_engine] Starting global step 403 (accumulating 32 micro-batches)... [2026-10-20 10:43:45][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 10:44:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4480 [2026-10-20 10:44:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1276 [2026-10-20 10:44:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1490 [2026-10-20 10:44:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3519 [2026-10-20 10:44:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9238 [2026-10-20 10:44:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5021 [2026-10-20 10:44:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2768 [2026-10-20 10:44:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8809 [2026-10-20 10:44:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1482 [2026-10-20 10:44:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0535 [2026-10-20 10:44:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1345 [2026-10-20 10:44:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5465 [2026-10-20 10:44:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0851 [2026-10-20 10:44:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2484 [2026-10-20 10:44:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3784 [2026-10-20 10:44:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0645 [2026-10-20 10:45:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4073 [2026-10-20 10:45:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9276 [2026-10-20 10:45:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3979 [2026-10-20 10:45:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3348 [2026-10-20 10:45:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3693 [2026-10-20 10:45:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8019 [2026-10-20 10:45:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2147 [2026-10-20 10:45:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7052 [2026-10-20 10:45:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2271 [2026-10-20 10:45:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7300 [2026-10-20 10:45:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0859 [2026-10-20 10:45:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6245 [2026-10-20 10:46:04][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,540 | MFU: 1.8% [2026-10-20 10:46:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0475 [2026-10-20 10:46:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1058 [2026-10-20 10:46:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0517 [2026-10-20 10:46:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0908 [2026-10-20 10:46:09][INFO][training.pretrain_engine] [Ph2|Step 403] loss=6.2402 | ppl=512.97 | aux=0.02512 | lr=2.62e-04 | gnorm=1.219 | tok/s=29,175 | MFU=7.0% | step=143762ms | tokens=0.430B | mem=4.2/32.7GB | cost=$211.73/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 05:29 (138.7h) | Phase3: SKIPPED | SFT: Oct 26 10:15 (143.5h) | DPO (Final): Oct 26 10:49 (144.1h) [2026-10-20 10:46:09][INFO][training.pretrain_engine] Starting global step 404 (accumulating 32 micro-batches)... [2026-10-20 10:46:09][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 10:46:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2540 [2026-10-20 10:46:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0976 [2026-10-20 10:46:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2081 [2026-10-20 10:46:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2812 [2026-10-20 10:46:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3512 [2026-10-20 10:46:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3532 [2026-10-20 10:46:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6056 [2026-10-20 10:46:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9587 [2026-10-20 10:47:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8327 [2026-10-20 10:47:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2266 [2026-10-20 10:47:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2280 [2026-10-20 10:47:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0274 [2026-10-20 10:47:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4779 [2026-10-20 10:47:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2459 [2026-10-20 10:47:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8273 [2026-10-20 10:47:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1617 [2026-10-20 10:47:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3413 [2026-10-20 10:47:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1366 [2026-10-20 10:47:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5412 [2026-10-20 10:47:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4089 [2026-10-20 10:47:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7717 [2026-10-20 10:47:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5825 [2026-10-20 10:47:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7263 [2026-10-20 10:47:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7856 [2026-10-20 10:48:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8472 [2026-10-20 10:48:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4571 [2026-10-20 10:48:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1516 [2026-10-20 10:48:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5485 [2026-10-20 10:48:28][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,541 | MFU: 1.8% [2026-10-20 10:48:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4934 [2026-10-20 10:48:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9679 [2026-10-20 10:48:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7229 [2026-10-20 10:48:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3804 [2026-10-20 10:48:32][INFO][training.pretrain_engine] [Ph2|Step 404] loss=6.1607 | ppl=473.77 | aux=0.02513 | lr=2.63e-04 | gnorm=0.895 | tok/s=29,174 | MFU=7.0% | step=143767ms | tokens=0.431B | mem=4.2/32.7GB | cost=$212.25/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 05:31 (138.7h) | Phase3: SKIPPED | SFT: Oct 26 10:17 (143.5h) | DPO (Final): Oct 26 10:51 (144.0h) [2026-10-20 10:48:32][INFO][training.pretrain_engine] Starting global step 405 (accumulating 32 micro-batches)... [2026-10-20 10:48:32][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 10:48:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9598 [2026-10-20 10:48:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7130 [2026-10-20 10:48:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6340 [2026-10-20 10:48:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7493 [2026-10-20 10:49:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7832 [2026-10-20 10:49:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9806 [2026-10-20 10:49:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4235 [2026-10-20 10:49:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6325 [2026-10-20 10:49:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1857 [2026-10-20 10:49:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8524 [2026-10-20 10:49:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2064 [2026-10-20 10:49:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3381 [2026-10-20 10:49:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7696 [2026-10-20 10:49:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2471 [2026-10-20 10:49:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9300 [2026-10-20 10:49:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2246 [2026-10-20 10:50:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3124 [2026-10-20 10:50:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5481 [2026-10-20 10:50:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0049 [2026-10-20 10:50:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3286 [2026-10-20 10:50:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6728 [2026-10-20 10:50:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0376 [2026-10-20 10:50:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5190 [2026-10-20 10:50:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4706 [2026-10-20 10:50:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0815 [2026-10-20 10:50:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9697 [2026-10-20 10:50:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9805 [2026-10-20 10:50:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9104 [2026-10-20 10:50:51][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,551 | MFU: 1.8% [2026-10-20 10:50:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2515 [2026-10-20 10:50:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6163 [2026-10-20 10:50:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4810 [2026-10-20 10:50:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9393 [2026-10-20 10:50:56][INFO][training.pretrain_engine] [Ph2|Step 405] loss=6.3003 | ppl=544.74 | aux=0.02513 | lr=2.63e-04 | gnorm=1.008 | tok/s=29,173 | MFU=7.0% | step=143772ms | tokens=0.432B | mem=4.2/32.7GB | cost=$212.78/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 05:33 (138.7h) | Phase3: SKIPPED | SFT: Oct 26 10:19 (143.5h) | DPO (Final): Oct 26 10:53 (144.0h) [2026-10-20 10:50:59][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step405 | Tokens: 0.432B | Spent: $212.79 | Remaining: $187.21 [2026-10-20 10:50:59][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step405 [2026-10-20 10:50:59][INFO][training.pretrain_engine] Starting global step 406 (accumulating 32 micro-batches)... [2026-10-20 10:50:59][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 10:50:59][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 10:50:59][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 10:50:59][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 10:50:59][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step405 to Phase-Technologies/forge-500m-pretrain (path: phase2_step405)... [2026-10-20 10:50:59][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 10:51:09][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 10:51:09][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step405 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 10:51:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3371 [2026-10-20 10:51:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3358 [2026-10-20 10:51:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2845 [2026-10-20 10:51:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2269 [2026-10-20 10:51:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9627 [2026-10-20 10:51:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5477 [2026-10-20 10:51:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6554 [2026-10-20 10:51:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2705 [2026-10-20 10:51:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1346 [2026-10-20 10:51:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7150 [2026-10-20 10:51:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2084 [2026-10-20 10:51:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3813 [2026-10-20 10:52:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8846 [2026-10-20 10:52:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8538 [2026-10-20 10:52:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1883 [2026-10-20 10:52:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2746 [2026-10-20 10:52:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3783 [2026-10-20 10:52:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2777 [2026-10-20 10:52:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3283 [2026-10-20 10:52:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3379 [2026-10-20 10:52:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9860 [2026-10-20 10:52:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8893 [2026-10-20 10:52:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3847 [2026-10-20 10:52:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0455 [2026-10-20 10:53:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0616 [2026-10-20 10:53:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8944 [2026-10-20 10:53:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0043 [2026-10-20 10:53:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0715 [2026-10-20 10:53:18][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,539 | MFU: 1.8% [2026-10-20 10:53:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9339 [2026-10-20 10:53:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5335 [2026-10-20 10:53:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8899 [2026-10-20 10:53:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5241 [2026-10-20 10:53:23][INFO][training.pretrain_engine] [Ph2|Step 406] loss=6.2722 | ppl=529.64 | aux=0.02514 | lr=2.64e-04 | gnorm=1.328 | tok/s=29,172 | MFU=7.0% | step=143779ms | tokens=0.433B | mem=4.2/32.7GB | cost=$213.31/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 05:35 (138.7h) | Phase3: SKIPPED | SFT: Oct 26 10:21 (143.5h) | DPO (Final): Oct 26 10:55 (144.0h) [2026-10-20 10:53:23][INFO][training.pretrain_engine] Starting global step 407 (accumulating 32 micro-batches)... [2026-10-20 10:53:23][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 10:53:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2280 [2026-10-20 10:53:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1995 [2026-10-20 10:53:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4223 [2026-10-20 10:53:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8725 [2026-10-20 10:53:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5143 [2026-10-20 10:53:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2149 [2026-10-20 10:53:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4269 [2026-10-20 10:53:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5391 [2026-10-20 10:54:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5092 [2026-10-20 10:54:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0223 [2026-10-20 10:54:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8509 [2026-10-20 10:54:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8291 [2026-10-20 10:54:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9592 [2026-10-20 10:54:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4109 [2026-10-20 10:54:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7878 [2026-10-20 10:54:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5835 [2026-10-20 10:54:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6984 [2026-10-20 10:54:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7460 [2026-10-20 10:54:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2779 [2026-10-20 10:54:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1785 [2026-10-20 10:55:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7968 [2026-10-20 10:55:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5361 [2026-10-20 10:55:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7863 [2026-10-20 10:55:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8754 [2026-10-20 10:55:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8055 [2026-10-20 10:55:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3156 [2026-10-20 10:55:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9596 [2026-10-20 10:55:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7629 [2026-10-20 10:55:42][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,522 | MFU: 1.8% [2026-10-20 10:55:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3240 [2026-10-20 10:55:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3893 [2026-10-20 10:55:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5914 [2026-10-20 10:55:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9082 [2026-10-20 10:55:47][INFO][training.pretrain_engine] [Ph2|Step 407] loss=6.2729 | ppl=529.99 | aux=0.02513 | lr=2.64e-04 | gnorm=1.055 | tok/s=29,169 | MFU=7.0% | step=143793ms | tokens=0.434B | mem=4.2/32.7GB | cost=$213.84/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 05:38 (138.7h) | Phase3: SKIPPED | SFT: Oct 26 10:24 (143.5h) | DPO (Final): Oct 26 10:58 (144.0h) [2026-10-20 10:55:47][INFO][training.pretrain_engine] Starting global step 408 (accumulating 32 micro-batches)... [2026-10-20 10:55:47][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 10:56:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5501 [2026-10-20 10:56:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1621 [2026-10-20 10:56:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6653 [2026-10-20 10:56:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3316 [2026-10-20 10:56:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8888 [2026-10-20 10:56:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1144 [2026-10-20 10:56:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9830 [2026-10-20 10:56:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9956 [2026-10-20 10:56:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1782 [2026-10-20 10:56:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8489 [2026-10-20 10:56:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0562 [2026-10-20 10:56:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6439 [2026-10-20 10:56:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6414 [2026-10-20 10:56:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0503 [2026-10-20 10:56:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1226 [2026-10-20 10:56:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3578 [2026-10-20 10:57:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2511 [2026-10-20 10:57:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0650 [2026-10-20 10:57:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3982 [2026-10-20 10:57:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0864 [2026-10-20 10:57:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1109 [2026-10-20 10:57:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2564 [2026-10-20 10:57:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1798 [2026-10-20 10:57:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6223 [2026-10-20 10:57:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0181 [2026-10-20 10:57:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6869 [2026-10-20 10:57:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0058 [2026-10-20 10:57:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2915 [2026-10-20 10:58:07][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,511 | MFU: 1.8% [2026-10-20 10:58:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5449 [2026-10-20 10:58:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2154 [2026-10-20 10:58:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6312 [2026-10-20 10:58:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1032 [2026-10-20 10:58:11][INFO][training.pretrain_engine] [Ph2|Step 408] loss=6.1677 | ppl=477.09 | aux=0.02513 | lr=2.65e-04 | gnorm=1.102 | tok/s=29,168 | MFU=7.0% | step=143800ms | tokens=0.435B | mem=4.2/32.7GB | cost=$214.37/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 05:40 (138.7h) | Phase3: SKIPPED | SFT: Oct 26 10:26 (143.5h) | DPO (Final): Oct 26 11:00 (144.0h) [2026-10-20 10:58:11][INFO][training.pretrain_engine] Starting global step 409 (accumulating 32 micro-batches)... [2026-10-20 10:58:11][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 10:58:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4449 [2026-10-20 10:58:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4717 [2026-10-20 10:58:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3982 [2026-10-20 10:58:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8269 [2026-10-20 10:58:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5883 [2026-10-20 10:58:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1044 [2026-10-20 10:58:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1282 [2026-10-20 10:58:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3684 [2026-10-20 10:59:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1147 [2026-10-20 10:59:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9620 [2026-10-20 10:59:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3318 [2026-10-20 10:59:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5540 [2026-10-20 10:59:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0835 [2026-10-20 10:59:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9219 [2026-10-20 10:59:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1528 [2026-10-20 10:59:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4057 [2026-10-20 10:59:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3304 [2026-10-20 10:59:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1126 [2026-10-20 10:59:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4328 [2026-10-20 10:59:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1929 [2026-10-20 10:59:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1881 [2026-10-20 10:59:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1946 [2026-10-20 10:59:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1157 [2026-10-20 10:59:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0134 [2026-10-20 11:00:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3272 [2026-10-20 11:00:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7206 [2026-10-20 11:00:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9126 [2026-10-20 11:00:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9140 [2026-10-20 11:00:30][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,565 | MFU: 1.8% [2026-10-20 11:00:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6276 [2026-10-20 11:00:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3350 [2026-10-20 11:00:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2901 [2026-10-20 11:00:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3467 [2026-10-20 11:00:35][INFO][training.pretrain_engine] [Ph2|Step 409] loss=6.2966 | ppl=542.71 | aux=0.02511 | lr=2.66e-04 | gnorm=0.824 | tok/s=29,168 | MFU=7.0% | step=143800ms | tokens=0.436B | mem=4.2/32.7GB | cost=$214.89/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 05:42 (138.7h) | Phase3: SKIPPED | SFT: Oct 26 10:28 (143.5h) | DPO (Final): Oct 26 11:02 (144.0h) [2026-10-20 11:00:35][INFO][training.pretrain_engine] Starting global step 410 (accumulating 32 micro-batches)... [2026-10-20 11:00:35][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 11:00:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1643 [2026-10-20 11:00:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2497 [2026-10-20 11:00:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0975 [2026-10-20 11:00:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8679 [2026-10-20 11:01:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3159 [2026-10-20 11:01:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2444 [2026-10-20 11:01:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2694 [2026-10-20 11:01:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0044 [2026-10-20 11:01:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3331 [2026-10-20 11:01:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5326 [2026-10-20 11:01:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3387 [2026-10-20 11:01:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4318 [2026-10-20 11:01:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.1013 [2026-10-20 11:01:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8742 [2026-10-20 11:01:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5805 [2026-10-20 11:01:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5506 [2026-10-20 11:02:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4365 [2026-10-20 11:02:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3265 [2026-10-20 11:02:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5408 [2026-10-20 11:02:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1471 [2026-10-20 11:02:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6781 [2026-10-20 11:02:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7177 [2026-10-20 11:02:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1704 [2026-10-20 11:02:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4156 [2026-10-20 11:02:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7862 [2026-10-20 11:02:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4027 [2026-10-20 11:02:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0096 [2026-10-20 11:02:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7372 [2026-10-20 11:02:54][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,545 | MFU: 1.8% [2026-10-20 11:02:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5986 [2026-10-20 11:02:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3814 [2026-10-20 11:02:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5574 [2026-10-20 11:02:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6595 [2026-10-20 11:02:58][INFO][training.pretrain_engine] [Ph2|Step 410] loss=6.2449 | ppl=515.38 | aux=0.02511 | lr=2.66e-04 | gnorm=0.914 | tok/s=29,167 | MFU=7.0% | step=143802ms | tokens=0.437B | mem=4.2/32.7GB | cost=$215.42/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 05:44 (138.7h) | Phase3: SKIPPED | SFT: Oct 26 10:30 (143.5h) | DPO (Final): Oct 26 11:04 (144.0h) [2026-10-20 11:02:58][INFO][training.pretrain_engine] Starting global step 411 (accumulating 32 micro-batches)... [2026-10-20 11:02:58][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 11:03:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0640 [2026-10-20 11:03:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4219 [2026-10-20 11:03:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6779 [2026-10-20 11:03:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2307 [2026-10-20 11:03:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2400 [2026-10-20 11:03:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6737 [2026-10-20 11:03:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4396 [2026-10-20 11:03:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1435 [2026-10-20 11:03:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2885 [2026-10-20 11:03:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1133 [2026-10-20 11:03:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9524 [2026-10-20 11:03:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1888 [2026-10-20 11:04:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9423 [2026-10-20 11:04:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9862 [2026-10-20 11:04:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1105 [2026-10-20 11:04:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1097 [2026-10-20 11:04:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9950 [2026-10-20 11:04:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3570 [2026-10-20 11:04:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0564 [2026-10-20 11:04:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2909 [2026-10-20 11:04:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2593 [2026-10-20 11:04:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7319 [2026-10-20 11:04:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1645 [2026-10-20 11:04:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0014 [2026-10-20 11:05:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9324 [2026-10-20 11:05:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2379 [2026-10-20 11:05:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4930 [2026-10-20 11:05:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6010 [2026-10-20 11:05:17][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,546 | MFU: 1.8% [2026-10-20 11:05:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5550 [2026-10-20 11:05:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5678 [2026-10-20 11:05:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3887 [2026-10-20 11:05:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6157 [2026-10-20 11:05:22][INFO][training.pretrain_engine] [Ph2|Step 411] loss=6.2128 | ppl=499.11 | aux=0.02511 | lr=2.67e-04 | gnorm=0.789 | tok/s=29,167 | MFU=7.0% | step=143801ms | tokens=0.438B | mem=4.2/32.7GB | cost=$215.94/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 05:46 (138.7h) | Phase3: SKIPPED | SFT: Oct 26 10:31 (143.4h) | DPO (Final): Oct 26 11:06 (144.0h) [2026-10-20 11:05:22][INFO][training.pretrain_engine] Starting global step 412 (accumulating 32 micro-batches)... [2026-10-20 11:05:22][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 11:05:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3527 [2026-10-20 11:05:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1683 [2026-10-20 11:05:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0091 [2026-10-20 11:05:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5416 [2026-10-20 11:05:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.4299 [2026-10-20 11:05:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2694 [2026-10-20 11:05:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6315 [2026-10-20 11:05:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6777 [2026-10-20 11:06:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0523 [2026-10-20 11:06:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2498 [2026-10-20 11:06:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8252 [2026-10-20 11:06:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1288 [2026-10-20 11:06:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3273 [2026-10-20 11:06:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9835 [2026-10-20 11:06:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9428 [2026-10-20 11:06:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7496 [2026-10-20 11:06:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4154 [2026-10-20 11:06:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8122 [2026-10-20 11:06:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4148 [2026-10-20 11:06:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1971 [2026-10-20 11:07:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2260 [2026-10-20 11:07:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0701 [2026-10-20 11:07:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1730 [2026-10-20 11:07:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8532 [2026-10-20 11:07:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3550 [2026-10-20 11:07:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9483 [2026-10-20 11:07:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1339 [2026-10-20 11:07:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4755 [2026-10-20 11:07:41][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,528 | MFU: 1.8% [2026-10-20 11:07:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2689 [2026-10-20 11:07:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6491 [2026-10-20 11:07:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1664 [2026-10-20 11:07:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0446 [2026-10-20 11:07:46][INFO][training.pretrain_engine] [Ph2|Step 412] loss=6.1817 | ppl=483.80 | aux=0.02514 | lr=2.67e-04 | gnorm=0.879 | tok/s=29,167 | MFU=7.0% | step=143805ms | tokens=0.439B | mem=4.2/32.7GB | cost=$216.47/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 05:48 (138.7h) | Phase3: SKIPPED | SFT: Oct 26 10:33 (143.4h) | DPO (Final): Oct 26 11:08 (144.0h) [2026-10-20 11:07:46][INFO][training.pretrain_engine] Starting global step 413 (accumulating 32 micro-batches)... [2026-10-20 11:07:46][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 11:08:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1731 [2026-10-20 11:08:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1667 [2026-10-20 11:08:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9753 [2026-10-20 11:08:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2641 [2026-10-20 11:08:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1366 [2026-10-20 11:08:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3461 [2026-10-20 11:08:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5393 [2026-10-20 11:08:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8882 [2026-10-20 11:08:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2686 [2026-10-20 11:08:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3395 [2026-10-20 11:08:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4206 [2026-10-20 11:08:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2296 [2026-10-20 11:08:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1022 [2026-10-20 11:08:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9001 [2026-10-20 11:08:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4168 [2026-10-20 11:08:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7798 [2026-10-20 11:09:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8478 [2026-10-20 11:09:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6290 [2026-10-20 11:09:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5145 [2026-10-20 11:09:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4976 [2026-10-20 11:09:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6946 [2026-10-20 11:09:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1236 [2026-10-20 11:09:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8641 [2026-10-20 11:09:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6995 [2026-10-20 11:09:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7060 [2026-10-20 11:09:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5776 [2026-10-20 11:09:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0172 [2026-10-20 11:09:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1658 [2026-10-20 11:10:05][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,515 | MFU: 1.8% [2026-10-20 11:10:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4660 [2026-10-20 11:10:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3303 [2026-10-20 11:10:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7761 [2026-10-20 11:10:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3348 [2026-10-20 11:10:10][INFO][training.pretrain_engine] [Ph2|Step 413] loss=6.1660 | ppl=476.28 | aux=0.02512 | lr=2.68e-04 | gnorm=0.867 | tok/s=29,165 | MFU=7.0% | step=143812ms | tokens=0.440B | mem=4.2/32.7GB | cost=$216.99/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 05:50 (138.7h) | Phase3: SKIPPED | SFT: Oct 26 10:36 (143.4h) | DPO (Final): Oct 26 11:10 (144.0h) [2026-10-20 11:10:10][INFO][training.pretrain_engine] Starting global step 414 (accumulating 32 micro-batches)... [2026-10-20 11:10:10][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 11:10:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4419 [2026-10-20 11:10:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2566 [2026-10-20 11:10:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1473 [2026-10-20 11:10:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7007 [2026-10-20 11:10:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1281 [2026-10-20 11:10:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6175 [2026-10-20 11:10:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9681 [2026-10-20 11:10:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0183 [2026-10-20 11:11:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8180 [2026-10-20 11:11:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0256 [2026-10-20 11:11:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0893 [2026-10-20 11:11:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3038 [2026-10-20 11:11:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0772 [2026-10-20 11:11:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4880 [2026-10-20 11:11:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2451 [2026-10-20 11:11:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1026 [2026-10-20 11:11:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1738 [2026-10-20 11:11:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5371 [2026-10-20 11:11:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8786 [2026-10-20 11:11:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1582 [2026-10-20 11:11:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0523 [2026-10-20 11:11:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2013 [2026-10-20 11:11:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3361 [2026-10-20 11:11:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4706 [2026-10-20 11:12:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9867 [2026-10-20 11:12:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0817 [2026-10-20 11:12:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9473 [2026-10-20 11:12:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9237 [2026-10-20 11:12:29][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,544 | MFU: 1.8% [2026-10-20 11:12:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0444 [2026-10-20 11:12:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1863 [2026-10-20 11:12:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9346 [2026-10-20 11:12:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7103 [2026-10-20 11:12:33][INFO][training.pretrain_engine] [Ph2|Step 414] loss=6.1848 | ppl=485.32 | aux=0.02511 | lr=2.68e-04 | gnorm=0.949 | tok/s=29,165 | MFU=7.0% | step=143811ms | tokens=0.441B | mem=4.2/32.7GB | cost=$217.52/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 05:52 (138.7h) | Phase3: SKIPPED | SFT: Oct 26 10:37 (143.4h) | DPO (Final): Oct 26 11:12 (144.0h) [2026-10-20 11:12:37][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step414 | Tokens: 0.441B | Spent: $217.53 | Remaining: $182.47 [2026-10-20 11:12:37][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step414 [2026-10-20 11:12:37][INFO][training.pretrain_engine] Starting global step 415 (accumulating 32 micro-batches)... [2026-10-20 11:12:37][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 11:12:37][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 11:12:37][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 11:12:37][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 11:12:37][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step414 to Phase-Technologies/forge-500m-pretrain (path: phase2_step414)... [2026-10-20 11:12:37][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 11:12:47][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 11:12:47][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step414 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 11:12:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7534 [2026-10-20 11:12:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8750 [2026-10-20 11:12:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0202 [2026-10-20 11:12:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5463 [2026-10-20 11:13:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0212 [2026-10-20 11:13:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0191 [2026-10-20 11:13:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2101 [2026-10-20 11:13:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0254 [2026-10-20 11:13:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0797 [2026-10-20 11:13:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1044 [2026-10-20 11:13:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3303 [2026-10-20 11:13:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5143 [2026-10-20 11:13:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0437 [2026-10-20 11:13:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2294 [2026-10-20 11:13:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0200 [2026-10-20 11:13:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0766 [2026-10-20 11:14:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1731 [2026-10-20 11:14:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3718 [2026-10-20 11:14:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3020 [2026-10-20 11:14:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5970 [2026-10-20 11:14:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8913 [2026-10-20 11:14:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9088 [2026-10-20 11:14:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7552 [2026-10-20 11:14:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9795 [2026-10-20 11:14:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0168 [2026-10-20 11:14:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4144 [2026-10-20 11:14:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3333 [2026-10-20 11:14:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9247 [2026-10-20 11:14:56][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,523 | MFU: 1.8% [2026-10-20 11:15:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4165 [2026-10-20 11:15:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3044 [2026-10-20 11:15:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2413 [2026-10-20 11:15:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2679 [2026-10-20 11:15:01][INFO][training.pretrain_engine] [Ph2|Step 415] loss=6.2433 | ppl=514.56 | aux=0.02512 | lr=2.69e-04 | gnorm=1.297 | tok/s=29,165 | MFU=7.0% | step=143813ms | tokens=0.442B | mem=4.2/32.7GB | cost=$218.06/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 05:54 (138.7h) | Phase3: SKIPPED | SFT: Oct 26 10:39 (143.4h) | DPO (Final): Oct 26 11:14 (144.0h) [2026-10-20 11:15:01][INFO][training.pretrain_engine] Starting global step 416 (accumulating 32 micro-batches)... [2026-10-20 11:15:01][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 11:15:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6709 [2026-10-20 11:15:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3442 [2026-10-20 11:15:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1662 [2026-10-20 11:15:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6986 [2026-10-20 11:15:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8000 [2026-10-20 11:15:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6259 [2026-10-20 11:15:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9076 [2026-10-20 11:15:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1462 [2026-10-20 11:15:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1928 [2026-10-20 11:15:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3333 [2026-10-20 11:15:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1983 [2026-10-20 11:15:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5544 [2026-10-20 11:16:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0933 [2026-10-20 11:16:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9629 [2026-10-20 11:16:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0149 [2026-10-20 11:16:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4646 [2026-10-20 11:16:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8443 [2026-10-20 11:16:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2656 [2026-10-20 11:16:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0928 [2026-10-20 11:16:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9959 [2026-10-20 11:16:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1973 [2026-10-20 11:16:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3479 [2026-10-20 11:16:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8188 [2026-10-20 11:16:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1857 [2026-10-20 11:17:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1710 [2026-10-20 11:17:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0517 [2026-10-20 11:17:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4177 [2026-10-20 11:17:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1907 [2026-10-20 11:17:20][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,516 | MFU: 1.8% [2026-10-20 11:17:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4118 [2026-10-20 11:17:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5303 [2026-10-20 11:17:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0826 [2026-10-20 11:17:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6006 [2026-10-20 11:17:25][INFO][training.pretrain_engine] [Ph2|Step 416] loss=6.1540 | ppl=470.59 | aux=0.02513 | lr=2.70e-04 | gnorm=0.934 | tok/s=29,165 | MFU=7.0% | step=143813ms | tokens=0.444B | mem=4.2/32.7GB | cost=$218.58/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 05:55 (138.6h) | Phase3: SKIPPED | SFT: Oct 26 10:41 (143.4h) | DPO (Final): Oct 26 11:15 (144.0h) [2026-10-20 11:17:25][INFO][training.pretrain_engine] Starting global step 417 (accumulating 32 micro-batches)... [2026-10-20 11:17:25][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 11:17:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0045 [2026-10-20 11:17:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1237 [2026-10-20 11:17:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8657 [2026-10-20 11:17:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1153 [2026-10-20 11:18:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5348 [2026-10-20 11:18:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0698 [2026-10-20 11:18:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8620 [2026-10-20 11:18:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0232 [2026-10-20 11:18:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9216 [2026-10-20 11:18:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2401 [2026-10-20 11:18:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1051 [2026-10-20 11:18:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0514 [2026-10-20 11:18:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6012 [2026-10-20 11:18:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9730 [2026-10-20 11:18:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9136 [2026-10-20 11:18:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4662 [2026-10-20 11:18:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1809 [2026-10-20 11:18:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0430 [2026-10-20 11:18:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9583 [2026-10-20 11:18:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7737 [2026-10-20 11:19:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7395 [2026-10-20 11:19:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9866 [2026-10-20 11:19:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9701 [2026-10-20 11:19:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6847 [2026-10-20 11:19:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9966 [2026-10-20 11:19:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6347 [2026-10-20 11:19:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9650 [2026-10-20 11:19:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8866 [2026-10-20 11:19:44][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,554 | MFU: 1.8% [2026-10-20 11:19:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6306 [2026-10-20 11:19:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9860 [2026-10-20 11:19:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6281 [2026-10-20 11:19:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9182 [2026-10-20 11:19:48][INFO][training.pretrain_engine] [Ph2|Step 417] loss=6.1386 | ppl=463.42 | aux=0.02512 | lr=2.70e-04 | gnorm=0.695 | tok/s=29,166 | MFU=7.0% | step=143809ms | tokens=0.445B | mem=4.2/32.7GB | cost=$219.11/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 05:57 (138.6h) | Phase3: SKIPPED | SFT: Oct 26 10:43 (143.4h) | DPO (Final): Oct 26 11:17 (144.0h) [2026-10-20 11:19:48][INFO][training.pretrain_engine] Starting global step 418 (accumulating 32 micro-batches)... [2026-10-20 11:19:48][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 11:20:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2117 [2026-10-20 11:20:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8259 [2026-10-20 11:20:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2594 [2026-10-20 11:20:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3397 [2026-10-20 11:20:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6510 [2026-10-20 11:20:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6016 [2026-10-20 11:20:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6042 [2026-10-20 11:20:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1208 [2026-10-20 11:20:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9702 [2026-10-20 11:20:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5992 [2026-10-20 11:20:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8241 [2026-10-20 11:20:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9735 [2026-10-20 11:21:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2091 [2026-10-20 11:21:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4597 [2026-10-20 11:21:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3728 [2026-10-20 11:21:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2125 [2026-10-20 11:21:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7118 [2026-10-20 11:21:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2665 [2026-10-20 11:21:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3721 [2026-10-20 11:21:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5644 [2026-10-20 11:21:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2084 [2026-10-20 11:21:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3869 [2026-10-20 11:21:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9203 [2026-10-20 11:21:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4059 [2026-10-20 11:21:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0766 [2026-10-20 11:21:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5973 [2026-10-20 11:21:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2033 [2026-10-20 11:21:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1753 [2026-10-20 11:22:08][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,508 | MFU: 1.8% [2026-10-20 11:22:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2686 [2026-10-20 11:22:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6928 [2026-10-20 11:22:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1855 [2026-10-20 11:22:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1959 [2026-10-20 11:22:12][INFO][training.pretrain_engine] [Ph2|Step 418] loss=6.0894 | ppl=441.16 | aux=0.02512 | lr=2.71e-04 | gnorm=1.016 | tok/s=29,165 | MFU=7.0% | step=143812ms | tokens=0.446B | mem=4.2/32.7GB | cost=$219.64/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 05:59 (138.6h) | Phase3: SKIPPED | SFT: Oct 26 10:45 (143.4h) | DPO (Final): Oct 26 11:19 (144.0h) [2026-10-20 11:22:12][INFO][training.pretrain_engine] Starting global step 419 (accumulating 32 micro-batches)... [2026-10-20 11:22:12][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 11:22:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8572 [2026-10-20 11:22:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1301 [2026-10-20 11:22:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9157 [2026-10-20 11:22:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8988 [2026-10-20 11:22:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6705 [2026-10-20 11:22:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1523 [2026-10-20 11:22:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0516 [2026-10-20 11:22:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8938 [2026-10-20 11:23:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7853 [2026-10-20 11:23:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7359 [2026-10-20 11:23:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7449 [2026-10-20 11:23:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2966 [2026-10-20 11:23:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0554 [2026-10-20 11:23:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9145 [2026-10-20 11:23:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5147 [2026-10-20 11:23:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1387 [2026-10-20 11:23:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0650 [2026-10-20 11:23:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4689 [2026-10-20 11:23:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8064 [2026-10-20 11:23:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0435 [2026-10-20 11:24:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3262 [2026-10-20 11:24:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5761 [2026-10-20 11:24:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3161 [2026-10-20 11:24:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4733 [2026-10-20 11:24:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0219 [2026-10-20 11:24:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3438 [2026-10-20 11:24:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2150 [2026-10-20 11:24:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0960 [2026-10-20 11:24:31][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,556 | MFU: 1.8% [2026-10-20 11:24:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6163 [2026-10-20 11:24:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7667 [2026-10-20 11:24:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4631 [2026-10-20 11:24:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6174 [2026-10-20 11:24:36][INFO][training.pretrain_engine] [Ph2|Step 419] loss=6.0890 | ppl=440.97 | aux=0.02510 | lr=2.71e-04 | gnorm=0.969 | tok/s=29,166 | MFU=7.0% | step=143807ms | tokens=0.447B | mem=4.2/32.7GB | cost=$220.16/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 06:00 (138.6h) | Phase3: SKIPPED | SFT: Oct 26 10:46 (143.4h) | DPO (Final): Oct 26 11:20 (143.9h) [2026-10-20 11:24:36][INFO][training.pretrain_engine] Starting global step 420 (accumulating 32 micro-batches)... [2026-10-20 11:24:36][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 11:24:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8303 [2026-10-20 11:24:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3576 [2026-10-20 11:24:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0068 [2026-10-20 11:24:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7031 [2026-10-20 11:25:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2499 [2026-10-20 11:25:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1746 [2026-10-20 11:25:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0645 [2026-10-20 11:25:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5055 [2026-10-20 11:25:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9378 [2026-10-20 11:25:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0989 [2026-10-20 11:25:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3604 [2026-10-20 11:25:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2677 [2026-10-20 11:25:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1074 [2026-10-20 11:25:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0024 [2026-10-20 11:25:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3364 [2026-10-20 11:25:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1654 [2026-10-20 11:26:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0393 [2026-10-20 11:26:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1786 [2026-10-20 11:26:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9945 [2026-10-20 11:26:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2254 [2026-10-20 11:26:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1824 [2026-10-20 11:26:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3803 [2026-10-20 11:26:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3708 [2026-10-20 11:26:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1450 [2026-10-20 11:26:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0068 [2026-10-20 11:26:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7915 [2026-10-20 11:26:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1612 [2026-10-20 11:26:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8394 [2026-10-20 11:26:55][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,561 | MFU: 1.8% [2026-10-20 11:27:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2485 [2026-10-20 11:27:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2154 [2026-10-20 11:27:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4212 [2026-10-20 11:27:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5617 [2026-10-20 11:27:00][INFO][training.pretrain_engine] [Ph2|Step 420] loss=6.1775 | ppl=481.77 | aux=0.02511 | lr=2.72e-04 | gnorm=0.855 | tok/s=29,168 | MFU=7.0% | step=143800ms | tokens=0.448B | mem=4.2/32.7GB | cost=$220.69/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 06:02 (138.6h) | Phase3: SKIPPED | SFT: Oct 26 10:48 (143.4h) | DPO (Final): Oct 26 11:22 (143.9h) [2026-10-20 11:27:00][INFO][training.pretrain_engine] Starting global step 421 (accumulating 32 micro-batches)... [2026-10-20 11:27:00][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 11:27:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4458 [2026-10-20 11:27:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6503 [2026-10-20 11:27:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8291 [2026-10-20 11:27:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0774 [2026-10-20 11:27:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1932 [2026-10-20 11:27:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1651 [2026-10-20 11:27:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2138 [2026-10-20 11:27:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1044 [2026-10-20 11:27:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0551 [2026-10-20 11:27:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7393 [2026-10-20 11:27:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9761 [2026-10-20 11:27:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9629 [2026-10-20 11:28:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3319 [2026-10-20 11:28:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0860 [2026-10-20 11:28:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4502 [2026-10-20 11:28:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3413 [2026-10-20 11:28:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7783 [2026-10-20 11:28:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2860 [2026-10-20 11:28:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4169 [2026-10-20 11:28:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1105 [2026-10-20 11:28:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6129 [2026-10-20 11:28:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5991 [2026-10-20 11:28:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9754 [2026-10-20 11:28:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8768 [2026-10-20 11:29:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9476 [2026-10-20 11:29:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4719 [2026-10-20 11:29:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6345 [2026-10-20 11:29:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9580 [2026-10-20 11:29:18][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,562 | MFU: 1.8% [2026-10-20 11:29:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2616 [2026-10-20 11:29:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7068 [2026-10-20 11:29:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7918 [2026-10-20 11:29:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5909 [2026-10-20 11:29:23][INFO][training.pretrain_engine] [Ph2|Step 421] loss=6.1580 | ppl=472.47 | aux=0.02512 | lr=2.72e-04 | gnorm=0.996 | tok/s=29,169 | MFU=7.0% | step=143792ms | tokens=0.449B | mem=4.2/32.7GB | cost=$221.21/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 06:03 (138.6h) | Phase3: SKIPPED | SFT: Oct 26 10:49 (143.3h) | DPO (Final): Oct 26 11:23 (143.9h) [2026-10-20 11:29:23][INFO][training.pretrain_engine] Starting global step 422 (accumulating 32 micro-batches)... [2026-10-20 11:29:23][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 11:29:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5763 [2026-10-20 11:29:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3965 [2026-10-20 11:29:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4925 [2026-10-20 11:29:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4033 [2026-10-20 11:29:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4909 [2026-10-20 11:29:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1149 [2026-10-20 11:29:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6405 [2026-10-20 11:29:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6282 [2026-10-20 11:30:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0186 [2026-10-20 11:30:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2828 [2026-10-20 11:30:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7033 [2026-10-20 11:30:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2805 [2026-10-20 11:30:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4894 [2026-10-20 11:30:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7478 [2026-10-20 11:30:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1831 [2026-10-20 11:30:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0472 [2026-10-20 11:30:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1490 [2026-10-20 11:30:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6097 [2026-10-20 11:30:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3803 [2026-10-20 11:30:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2039 [2026-10-20 11:31:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7564 [2026-10-20 11:31:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6727 [2026-10-20 11:31:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8966 [2026-10-20 11:31:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0205 [2026-10-20 11:31:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9080 [2026-10-20 11:31:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2418 [2026-10-20 11:31:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3739 [2026-10-20 11:31:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9820 [2026-10-20 11:31:42][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,563 | MFU: 1.8% [2026-10-20 11:31:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9113 [2026-10-20 11:31:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1203 [2026-10-20 11:31:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0988 [2026-10-20 11:31:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6758 [2026-10-20 11:31:47][INFO][training.pretrain_engine] [Ph2|Step 422] loss=6.1057 | ppl=448.40 | aux=0.02509 | lr=2.73e-04 | gnorm=1.305 | tok/s=29,169 | MFU=7.0% | step=143791ms | tokens=0.450B | mem=4.2/32.7GB | cost=$221.74/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 06:05 (138.6h) | Phase3: SKIPPED | SFT: Oct 26 10:51 (143.3h) | DPO (Final): Oct 26 11:25 (143.9h) [2026-10-20 11:31:47][INFO][training.pretrain_engine] Starting global step 423 (accumulating 32 micro-batches)... [2026-10-20 11:31:47][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 11:32:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5548 [2026-10-20 11:32:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0783 [2026-10-20 11:32:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3074 [2026-10-20 11:32:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5103 [2026-10-20 11:32:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7200 [2026-10-20 11:32:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0441 [2026-10-20 11:32:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3025 [2026-10-20 11:32:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1228 [2026-10-20 11:32:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0315 [2026-10-20 11:32:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2152 [2026-10-20 11:32:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3574 [2026-10-20 11:32:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1859 [2026-10-20 11:32:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2862 [2026-10-20 11:32:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6772 [2026-10-20 11:32:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1346 [2026-10-20 11:32:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2268 [2026-10-20 11:33:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1613 [2026-10-20 11:33:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7761 [2026-10-20 11:33:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0492 [2026-10-20 11:33:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8368 [2026-10-20 11:33:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4904 [2026-10-20 11:33:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1533 [2026-10-20 11:33:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3973 [2026-10-20 11:33:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3362 [2026-10-20 11:33:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5375 [2026-10-20 11:33:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2702 [2026-10-20 11:33:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1759 [2026-10-20 11:33:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6716 [2026-10-20 11:34:05][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,590 | MFU: 1.8% [2026-10-20 11:34:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3996 [2026-10-20 11:34:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6247 [2026-10-20 11:34:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4725 [2026-10-20 11:34:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2225 [2026-10-20 11:34:11][INFO][training.pretrain_engine] [Ph2|Step 423] loss=6.0929 | ppl=442.70 | aux=0.02510 | lr=2.73e-04 | gnorm=0.996 | tok/s=29,170 | MFU=7.0% | step=143786ms | tokens=0.451B | mem=4.2/32.7GB | cost=$222.26/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 06:06 (138.5h) | Phase3: SKIPPED | SFT: Oct 26 10:52 (143.3h) | DPO (Final): Oct 26 11:26 (143.9h) [2026-10-20 11:34:14][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step423 | Tokens: 0.451B | Spent: $222.27 | Remaining: $177.73 [2026-10-20 11:34:14][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step423 [2026-10-20 11:34:14][INFO][training.pretrain_engine] Starting global step 424 (accumulating 32 micro-batches)... [2026-10-20 11:34:14][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 11:34:14][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 11:34:14][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 11:34:14][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 11:34:14][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step423 to Phase-Technologies/forge-500m-pretrain (path: phase2_step423)... [2026-10-20 11:34:14][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 11:34:23][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 11:34:23][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step423 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 11:34:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1202 [2026-10-20 11:34:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1255 [2026-10-20 11:34:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7351 [2026-10-20 11:34:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6671 [2026-10-20 11:34:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6839 [2026-10-20 11:34:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6289 [2026-10-20 11:34:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6828 [2026-10-20 11:34:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9421 [2026-10-20 11:35:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0323 [2026-10-20 11:35:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9713 [2026-10-20 11:35:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7744 [2026-10-20 11:35:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1024 [2026-10-20 11:35:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0911 [2026-10-20 11:35:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9972 [2026-10-20 11:35:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8903 [2026-10-20 11:35:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8801 [2026-10-20 11:35:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2116 [2026-10-20 11:35:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4711 [2026-10-20 11:35:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0855 [2026-10-20 11:35:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5740 [2026-10-20 11:35:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5353 [2026-10-20 11:35:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4424 [2026-10-20 11:35:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5177 [2026-10-20 11:36:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8339 [2026-10-20 11:36:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0648 [2026-10-20 11:36:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1373 [2026-10-20 11:36:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8804 [2026-10-20 11:36:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1255 [2026-10-20 11:36:32][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,582 | MFU: 1.8% [2026-10-20 11:36:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2199 [2026-10-20 11:36:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3052 [2026-10-20 11:36:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0073 [2026-10-20 11:36:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6459 [2026-10-20 11:36:37][INFO][training.pretrain_engine] [Ph2|Step 424] loss=6.1330 | ppl=460.80 | aux=0.02510 | lr=2.74e-04 | gnorm=1.094 | tok/s=29,174 | MFU=7.0% | step=143771ms | tokens=0.452B | mem=4.2/32.7GB | cost=$222.79/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 06:07 (138.5h) | Phase3: SKIPPED | SFT: Oct 26 10:53 (143.3h) | DPO (Final): Oct 26 11:27 (143.9h) [2026-10-20 11:36:37][INFO][training.pretrain_engine] Starting global step 425 (accumulating 32 micro-batches)... [2026-10-20 11:36:37][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 11:36:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9837 [2026-10-20 11:36:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4626 [2026-10-20 11:36:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7546 [2026-10-20 11:36:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6930 [2026-10-20 11:37:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0820 [2026-10-20 11:37:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3013 [2026-10-20 11:37:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8499 [2026-10-20 11:37:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7258 [2026-10-20 11:37:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1434 [2026-10-20 11:37:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8038 [2026-10-20 11:37:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0707 [2026-10-20 11:37:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9984 [2026-10-20 11:37:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0209 [2026-10-20 11:37:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3367 [2026-10-20 11:37:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1285 [2026-10-20 11:37:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0744 [2026-10-20 11:38:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6577 [2026-10-20 11:38:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4654 [2026-10-20 11:38:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5468 [2026-10-20 11:38:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2277 [2026-10-20 11:38:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5073 [2026-10-20 11:38:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3275 [2026-10-20 11:38:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2388 [2026-10-20 11:38:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2092 [2026-10-20 11:38:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3068 [2026-10-20 11:38:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8369 [2026-10-20 11:38:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4356 [2026-10-20 11:38:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0687 [2026-10-20 11:38:55][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,588 | MFU: 1.8% [2026-10-20 11:39:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3408 [2026-10-20 11:39:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5974 [2026-10-20 11:39:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0038 [2026-10-20 11:39:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1201 [2026-10-20 11:39:00][INFO][training.pretrain_engine] [Ph2|Step 425] loss=6.2556 | ppl=520.94 | aux=0.02513 | lr=2.75e-04 | gnorm=1.242 | tok/s=29,174 | MFU=7.0% | step=143766ms | tokens=0.453B | mem=4.2/32.7GB | cost=$223.32/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 06:09 (138.5h) | Phase3: SKIPPED | SFT: Oct 26 10:55 (143.3h) | DPO (Final): Oct 26 11:29 (143.8h) [2026-10-20 11:39:00][INFO][training.pretrain_engine] Starting global step 426 (accumulating 32 micro-batches)... [2026-10-20 11:39:00][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 11:39:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2765 [2026-10-20 11:39:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5842 [2026-10-20 11:39:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1049 [2026-10-20 11:39:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1498 [2026-10-20 11:39:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1687 [2026-10-20 11:39:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1317 [2026-10-20 11:39:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9421 [2026-10-20 11:39:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9753 [2026-10-20 11:39:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4880 [2026-10-20 11:39:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7358 [2026-10-20 11:39:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2986 [2026-10-20 11:39:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7974 [2026-10-20 11:40:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2874 [2026-10-20 11:40:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4341 [2026-10-20 11:40:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8497 [2026-10-20 11:40:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4716 [2026-10-20 11:40:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1548 [2026-10-20 11:40:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0118 [2026-10-20 11:40:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4436 [2026-10-20 11:40:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2310 [2026-10-20 11:40:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3695 [2026-10-20 11:40:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6234 [2026-10-20 11:40:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1559 [2026-10-20 11:40:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1662 [2026-10-20 11:41:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2168 [2026-10-20 11:41:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7858 [2026-10-20 11:41:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4372 [2026-10-20 11:41:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1571 [2026-10-20 11:41:18][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,604 | MFU: 1.8% [2026-10-20 11:41:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1349 [2026-10-20 11:41:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6852 [2026-10-20 11:41:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3741 [2026-10-20 11:41:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0566 [2026-10-20 11:41:24][INFO][training.pretrain_engine] [Ph2|Step 426] loss=6.2324 | ppl=509.00 | aux=0.02512 | lr=2.75e-04 | gnorm=1.141 | tok/s=29,176 | MFU=7.0% | step=143760ms | tokens=0.454B | mem=4.2/32.7GB | cost=$223.84/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 06:10 (138.5h) | Phase3: SKIPPED | SFT: Oct 26 10:56 (143.3h) | DPO (Final): Oct 26 11:30 (143.8h) [2026-10-20 11:41:24][INFO][training.pretrain_engine] Starting global step 427 (accumulating 32 micro-batches)... [2026-10-20 11:41:24][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 11:41:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7586 [2026-10-20 11:41:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9384 [2026-10-20 11:41:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5424 [2026-10-20 11:41:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0194 [2026-10-20 11:42:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2975 [2026-10-20 11:42:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3080 [2026-10-20 11:42:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3662 [2026-10-20 11:42:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1209 [2026-10-20 11:42:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0196 [2026-10-20 11:42:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4537 [2026-10-20 11:42:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7395 [2026-10-20 11:42:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2449 [2026-10-20 11:42:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5870 [2026-10-20 11:42:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0094 [2026-10-20 11:42:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3032 [2026-10-20 11:42:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3382 [2026-10-20 11:42:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8297 [2026-10-20 11:42:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3561 [2026-10-20 11:42:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2905 [2026-10-20 11:42:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2834 [2026-10-20 11:43:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3213 [2026-10-20 11:43:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4541 [2026-10-20 11:43:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4211 [2026-10-20 11:43:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1122 [2026-10-20 11:43:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5466 [2026-10-20 11:43:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7874 [2026-10-20 11:43:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2497 [2026-10-20 11:43:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7427 [2026-10-20 11:43:43][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,535 | MFU: 1.8% [2026-10-20 11:43:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6529 [2026-10-20 11:43:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9290 [2026-10-20 11:43:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6692 [2026-10-20 11:43:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3477 [2026-10-20 11:43:48][INFO][training.pretrain_engine] [Ph2|Step 427] loss=6.1673 | ppl=476.88 | aux=0.02512 | lr=2.76e-04 | gnorm=1.211 | tok/s=29,175 | MFU=7.0% | step=143763ms | tokens=0.455B | mem=4.2/32.7GB | cost=$224.37/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 06:12 (138.5h) | Phase3: SKIPPED | SFT: Oct 26 10:58 (143.2h) | DPO (Final): Oct 26 11:32 (143.8h) [2026-10-20 11:43:48][INFO][training.pretrain_engine] Starting global step 428 (accumulating 32 micro-batches)... [2026-10-20 11:43:48][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 11:44:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7844 [2026-10-20 11:44:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4739 [2026-10-20 11:44:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8950 [2026-10-20 11:44:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8471 [2026-10-20 11:44:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0506 [2026-10-20 11:44:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0172 [2026-10-20 11:44:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6073 [2026-10-20 11:44:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5114 [2026-10-20 11:44:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8579 [2026-10-20 11:44:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0028 [2026-10-20 11:44:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8265 [2026-10-20 11:44:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5479 [2026-10-20 11:44:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9214 [2026-10-20 11:44:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9610 [2026-10-20 11:44:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7025 [2026-10-20 11:45:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8379 [2026-10-20 11:45:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0089 [2026-10-20 11:45:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1578 [2026-10-20 11:45:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2139 [2026-10-20 11:45:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1095 [2026-10-20 11:45:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7004 [2026-10-20 11:45:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0845 [2026-10-20 11:45:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3154 [2026-10-20 11:45:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4396 [2026-10-20 11:45:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8341 [2026-10-20 11:45:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6982 [2026-10-20 11:45:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9193 [2026-10-20 11:45:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9186 [2026-10-20 11:46:07][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,543 | MFU: 1.8% [2026-10-20 11:46:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9880 [2026-10-20 11:46:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7993 [2026-10-20 11:46:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9341 [2026-10-20 11:46:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5810 [2026-10-20 11:46:11][INFO][training.pretrain_engine] [Ph2|Step 428] loss=6.0705 | ppl=432.89 | aux=0.02513 | lr=2.76e-04 | gnorm=0.934 | tok/s=29,175 | MFU=7.0% | step=143763ms | tokens=0.456B | mem=4.2/32.7GB | cost=$224.90/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 06:14 (138.5h) | Phase3: SKIPPED | SFT: Oct 26 11:00 (143.2h) | DPO (Final): Oct 26 11:34 (143.8h) [2026-10-20 11:46:11][INFO][training.pretrain_engine] Starting global step 429 (accumulating 32 micro-batches)... [2026-10-20 11:46:11][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 11:46:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8977 [2026-10-20 11:46:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3399 [2026-10-20 11:46:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3845 [2026-10-20 11:46:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3723 [2026-10-20 11:46:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0018 [2026-10-20 11:46:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7925 [2026-10-20 11:46:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5644 [2026-10-20 11:46:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7754 [2026-10-20 11:47:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0457 [2026-10-20 11:47:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2015 [2026-10-20 11:47:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1224 [2026-10-20 11:47:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2176 [2026-10-20 11:47:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2225 [2026-10-20 11:47:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2048 [2026-10-20 11:47:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9233 [2026-10-20 11:47:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9434 [2026-10-20 11:47:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6227 [2026-10-20 11:47:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8339 [2026-10-20 11:47:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7260 [2026-10-20 11:47:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9770 [2026-10-20 11:47:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0507 [2026-10-20 11:47:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6474 [2026-10-20 11:47:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0941 [2026-10-20 11:47:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4136 [2026-10-20 11:48:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3846 [2026-10-20 11:48:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3224 [2026-10-20 11:48:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1195 [2026-10-20 11:48:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1511 [2026-10-20 11:48:30][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,540 | MFU: 1.8% [2026-10-20 11:48:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1520 [2026-10-20 11:48:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9742 [2026-10-20 11:48:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3403 [2026-10-20 11:48:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0222 [2026-10-20 11:48:35][INFO][training.pretrain_engine] [Ph2|Step 429] loss=6.1801 | ppl=483.03 | aux=0.02511 | lr=2.77e-04 | gnorm=1.492 | tok/s=29,175 | MFU=7.0% | step=143764ms | tokens=0.457B | mem=4.2/32.7GB | cost=$225.42/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 06:16 (138.5h) | Phase3: SKIPPED | SFT: Oct 26 11:02 (143.2h) | DPO (Final): Oct 26 11:36 (143.8h) [2026-10-20 11:48:35][INFO][training.pretrain_engine] Starting global step 430 (accumulating 32 micro-batches)... [2026-10-20 11:48:35][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 11:48:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2669 [2026-10-20 11:48:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2358 [2026-10-20 11:48:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0431 [2026-10-20 11:48:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3353 [2026-10-20 11:49:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2046 [2026-10-20 11:49:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4130 [2026-10-20 11:49:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8199 [2026-10-20 11:49:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9306 [2026-10-20 11:49:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2819 [2026-10-20 11:49:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1561 [2026-10-20 11:49:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9388 [2026-10-20 11:49:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8534 [2026-10-20 11:49:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9002 [2026-10-20 11:49:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1427 [2026-10-20 11:49:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1417 [2026-10-20 11:49:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4658 [2026-10-20 11:50:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0492 [2026-10-20 11:50:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0266 [2026-10-20 11:50:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9616 [2026-10-20 11:50:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3667 [2026-10-20 11:50:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5378 [2026-10-20 11:50:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4924 [2026-10-20 11:50:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7454 [2026-10-20 11:50:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2273 [2026-10-20 11:50:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0842 [2026-10-20 11:50:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0532 [2026-10-20 11:50:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2732 [2026-10-20 11:50:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8706 [2026-10-20 11:50:54][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,555 | MFU: 1.8% [2026-10-20 11:50:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8090 [2026-10-20 11:50:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1154 [2026-10-20 11:50:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9862 [2026-10-20 11:50:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8171 [2026-10-20 11:50:59][INFO][training.pretrain_engine] [Ph2|Step 430] loss=6.1262 | ppl=457.72 | aux=0.02511 | lr=2.77e-04 | gnorm=0.887 | tok/s=29,175 | MFU=7.0% | step=143762ms | tokens=0.458B | mem=4.2/32.7GB | cost=$225.95/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 06:18 (138.5h) | Phase3: SKIPPED | SFT: Oct 26 11:03 (143.2h) | DPO (Final): Oct 26 11:38 (143.8h) [2026-10-20 11:50:59][INFO][training.pretrain_engine] Starting global step 431 (accumulating 32 micro-batches)... [2026-10-20 11:50:59][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 11:51:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4859 [2026-10-20 11:51:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1781 [2026-10-20 11:51:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6308 [2026-10-20 11:51:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3541 [2026-10-20 11:51:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2458 [2026-10-20 11:51:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1969 [2026-10-20 11:51:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1702 [2026-10-20 11:51:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7087 [2026-10-20 11:51:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0070 [2026-10-20 11:51:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0013 [2026-10-20 11:51:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1745 [2026-10-20 11:51:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2200 [2026-10-20 11:52:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8669 [2026-10-20 11:52:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6992 [2026-10-20 11:52:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7200 [2026-10-20 11:52:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8825 [2026-10-20 11:52:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1238 [2026-10-20 11:52:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5416 [2026-10-20 11:52:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5569 [2026-10-20 11:52:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9191 [2026-10-20 11:52:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0545 [2026-10-20 11:52:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9874 [2026-10-20 11:52:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2057 [2026-10-20 11:52:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9462 [2026-10-20 11:53:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0700 [2026-10-20 11:53:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0889 [2026-10-20 11:53:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1770 [2026-10-20 11:53:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6203 [2026-10-20 11:53:17][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,551 | MFU: 1.8% [2026-10-20 11:53:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9673 [2026-10-20 11:53:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0034 [2026-10-20 11:53:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9460 [2026-10-20 11:53:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5862 [2026-10-20 11:53:22][INFO][training.pretrain_engine] [Ph2|Step 431] loss=6.1080 | ppl=449.46 | aux=0.02509 | lr=2.78e-04 | gnorm=1.430 | tok/s=29,177 | MFU=7.0% | step=143756ms | tokens=0.459B | mem=4.2/32.7GB | cost=$226.47/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 06:19 (138.4h) | Phase3: SKIPPED | SFT: Oct 26 11:05 (143.2h) | DPO (Final): Oct 26 11:39 (143.8h) [2026-10-20 11:53:22][INFO][training.pretrain_engine] Starting global step 432 (accumulating 32 micro-batches)... [2026-10-20 11:53:22][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 11:53:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4443 [2026-10-20 11:53:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0589 [2026-10-20 11:53:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2526 [2026-10-20 11:53:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9923 [2026-10-20 11:53:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7030 [2026-10-20 11:53:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1299 [2026-10-20 11:53:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7013 [2026-10-20 11:53:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3790 [2026-10-20 11:54:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7137 [2026-10-20 11:54:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7106 [2026-10-20 11:54:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5159 [2026-10-20 11:54:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1234 [2026-10-20 11:54:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3797 [2026-10-20 11:54:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3502 [2026-10-20 11:54:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6957 [2026-10-20 11:54:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8660 [2026-10-20 11:54:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9802 [2026-10-20 11:54:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0407 [2026-10-20 11:54:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0466 [2026-10-20 11:54:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9608 [2026-10-20 11:55:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6904 [2026-10-20 11:55:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3997 [2026-10-20 11:55:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2646 [2026-10-20 11:55:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6683 [2026-10-20 11:55:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3095 [2026-10-20 11:55:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0685 [2026-10-20 11:55:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3354 [2026-10-20 11:55:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1359 [2026-10-20 11:55:41][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,540 | MFU: 1.8% [2026-10-20 11:55:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9365 [2026-10-20 11:55:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7968 [2026-10-20 11:55:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4948 [2026-10-20 11:55:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1503 [2026-10-20 11:55:45][INFO][training.pretrain_engine] [Ph2|Step 432] loss=6.1571 | ppl=472.06 | aux=0.02510 | lr=2.79e-04 | gnorm=1.047 | tok/s=29,177 | MFU=7.0% | step=143756ms | tokens=0.460B | mem=4.2/32.7GB | cost=$226.99/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 06:21 (138.4h) | Phase3: SKIPPED | SFT: Oct 26 11:06 (143.2h) | DPO (Final): Oct 26 11:41 (143.8h) [2026-10-20 11:55:49][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step432 | Tokens: 0.460B | Spent: $227.01 | Remaining: $172.99 [2026-10-20 11:55:49][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step432 [2026-10-20 11:55:49][INFO][training.pretrain_engine] Starting global step 433 (accumulating 32 micro-batches)... [2026-10-20 11:55:49][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 11:55:49][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 11:55:49][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 11:55:49][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 11:55:49][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step432 to Phase-Technologies/forge-500m-pretrain (path: phase2_step432)... [2026-10-20 11:55:49][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 11:55:58][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 11:55:58][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step432 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 11:56:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8598 [2026-10-20 11:56:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0871 [2026-10-20 11:56:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8621 [2026-10-20 11:56:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1050 [2026-10-20 11:56:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8995 [2026-10-20 11:56:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3728 [2026-10-20 11:56:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9258 [2026-10-20 11:56:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0135 [2026-10-20 11:56:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0389 [2026-10-20 11:56:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1549 [2026-10-20 11:56:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9550 [2026-10-20 11:56:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0106 [2026-10-20 11:56:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4348 [2026-10-20 11:56:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6692 [2026-10-20 11:56:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3066 [2026-10-20 11:57:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5495 [2026-10-20 11:57:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0073 [2026-10-20 11:57:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0813 [2026-10-20 11:57:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3115 [2026-10-20 11:57:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2608 [2026-10-20 11:57:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0570 [2026-10-20 11:57:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4145 [2026-10-20 11:57:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8559 [2026-10-20 11:57:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5145 [2026-10-20 11:57:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4399 [2026-10-20 11:57:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7964 [2026-10-20 11:57:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6982 [2026-10-20 11:57:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1556 [2026-10-20 11:58:08][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,543 | MFU: 1.8% [2026-10-20 11:58:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9978 [2026-10-20 11:58:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4903 [2026-10-20 11:58:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0157 [2026-10-20 11:58:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3374 [2026-10-20 11:58:13][INFO][training.pretrain_engine] [Ph2|Step 433] loss=6.0890 | ppl=440.98 | aux=0.02510 | lr=2.79e-04 | gnorm=1.320 | tok/s=29,177 | MFU=7.0% | step=143752ms | tokens=0.461B | mem=4.2/32.7GB | cost=$227.53/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 06:22 (138.4h) | Phase3: SKIPPED | SFT: Oct 26 11:08 (143.2h) | DPO (Final): Oct 26 11:42 (143.7h) [2026-10-20 11:58:13][INFO][training.pretrain_engine] Starting global step 434 (accumulating 32 micro-batches)... [2026-10-20 11:58:13][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 11:58:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0349 [2026-10-20 11:58:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7771 [2026-10-20 11:58:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3700 [2026-10-20 11:58:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1662 [2026-10-20 11:58:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2305 [2026-10-20 11:58:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3109 [2026-10-20 11:58:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9348 [2026-10-20 11:58:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5126 [2026-10-20 11:59:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1257 [2026-10-20 11:59:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9424 [2026-10-20 11:59:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8631 [2026-10-20 11:59:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6814 [2026-10-20 11:59:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2266 [2026-10-20 11:59:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0061 [2026-10-20 11:59:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7324 [2026-10-20 11:59:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1319 [2026-10-20 11:59:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7480 [2026-10-20 11:59:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4598 [2026-10-20 11:59:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6409 [2026-10-20 11:59:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8824 [2026-10-20 12:00:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8941 [2026-10-20 12:00:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5739 [2026-10-20 12:00:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2432 [2026-10-20 12:00:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4095 [2026-10-20 12:00:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4351 [2026-10-20 12:00:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0532 [2026-10-20 12:00:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6356 [2026-10-20 12:00:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5198 [2026-10-20 12:00:32][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,506 | MFU: 1.8% [2026-10-20 12:00:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9840 [2026-10-20 12:00:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7242 [2026-10-20 12:00:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3899 [2026-10-20 12:00:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2763 [2026-10-20 12:00:37][INFO][training.pretrain_engine] [Ph2|Step 434] loss=6.0738 | ppl=434.33 | aux=0.02510 | lr=2.80e-04 | gnorm=0.906 | tok/s=29,176 | MFU=7.0% | step=143757ms | tokens=0.462B | mem=4.2/32.7GB | cost=$228.06/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 06:25 (138.4h) | Phase3: SKIPPED | SFT: Oct 26 11:10 (143.2h) | DPO (Final): Oct 26 11:44 (143.7h) [2026-10-20 12:00:37][INFO][training.pretrain_engine] Starting global step 435 (accumulating 32 micro-batches)... [2026-10-20 12:00:37][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 12:00:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6254 [2026-10-20 12:00:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2307 [2026-10-20 12:00:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9576 [2026-10-20 12:00:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1219 [2026-10-20 12:01:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1653 [2026-10-20 12:01:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7771 [2026-10-20 12:01:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0746 [2026-10-20 12:01:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1625 [2026-10-20 12:01:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7294 [2026-10-20 12:01:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7693 [2026-10-20 12:01:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8018 [2026-10-20 12:01:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8048 [2026-10-20 12:01:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5149 [2026-10-20 12:01:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7108 [2026-10-20 12:01:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5522 [2026-10-20 12:01:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9199 [2026-10-20 12:02:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4963 [2026-10-20 12:02:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9364 [2026-10-20 12:02:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9743 [2026-10-20 12:02:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1398 [2026-10-20 12:02:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9707 [2026-10-20 12:02:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3293 [2026-10-20 12:02:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8762 [2026-10-20 12:02:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6101 [2026-10-20 12:02:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9494 [2026-10-20 12:02:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5543 [2026-10-20 12:02:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1829 [2026-10-20 12:02:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2116 [2026-10-20 12:02:55][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,565 | MFU: 1.8% [2026-10-20 12:03:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4573 [2026-10-20 12:03:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1575 [2026-10-20 12:03:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2375 [2026-10-20 12:03:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3309 [2026-10-20 12:03:00][INFO][training.pretrain_engine] [Ph2|Step 435] loss=6.0780 | ppl=436.17 | aux=0.02511 | lr=2.80e-04 | gnorm=1.227 | tok/s=29,177 | MFU=7.0% | step=143754ms | tokens=0.463B | mem=4.2/32.7GB | cost=$228.58/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 06:26 (138.4h) | Phase3: SKIPPED | SFT: Oct 26 11:12 (143.2h) | DPO (Final): Oct 26 11:46 (143.7h) [2026-10-20 12:03:00][INFO][training.pretrain_engine] Starting global step 436 (accumulating 32 micro-batches)... [2026-10-20 12:03:00][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 12:03:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6061 [2026-10-20 12:03:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6006 [2026-10-20 12:03:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9431 [2026-10-20 12:03:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5243 [2026-10-20 12:03:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1211 [2026-10-20 12:03:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9696 [2026-10-20 12:03:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6628 [2026-10-20 12:03:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3339 [2026-10-20 12:03:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9490 [2026-10-20 12:03:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1267 [2026-10-20 12:03:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8403 [2026-10-20 12:03:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9170 [2026-10-20 12:04:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2571 [2026-10-20 12:04:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6859 [2026-10-20 12:04:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2691 [2026-10-20 12:04:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4290 [2026-10-20 12:04:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1227 [2026-10-20 12:04:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3132 [2026-10-20 12:04:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4076 [2026-10-20 12:04:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8220 [2026-10-20 12:04:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0199 [2026-10-20 12:04:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9331 [2026-10-20 12:04:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7746 [2026-10-20 12:04:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8973 [2026-10-20 12:05:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5294 [2026-10-20 12:05:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2839 [2026-10-20 12:05:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5437 [2026-10-20 12:05:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2824 [2026-10-20 12:05:18][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,590 | MFU: 1.8% [2026-10-20 12:05:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8542 [2026-10-20 12:05:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1093 [2026-10-20 12:05:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0603 [2026-10-20 12:05:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7774 [2026-10-20 12:05:24][INFO][training.pretrain_engine] [Ph2|Step 436] loss=6.0698 | ppl=432.59 | aux=0.02510 | lr=2.81e-04 | gnorm=0.965 | tok/s=29,179 | MFU=7.0% | step=143746ms | tokens=0.465B | mem=4.2/32.7GB | cost=$229.11/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 06:28 (138.4h) | Phase3: SKIPPED | SFT: Oct 26 11:13 (143.1h) | DPO (Final): Oct 26 11:47 (143.7h) [2026-10-20 12:05:24][INFO][training.pretrain_engine] Starting global step 437 (accumulating 32 micro-batches)... [2026-10-20 12:05:24][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 12:05:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3020 [2026-10-20 12:05:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2110 [2026-10-20 12:05:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0540 [2026-10-20 12:05:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2276 [2026-10-20 12:06:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1013 [2026-10-20 12:06:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0377 [2026-10-20 12:06:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8558 [2026-10-20 12:06:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2036 [2026-10-20 12:06:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3423 [2026-10-20 12:06:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2080 [2026-10-20 12:06:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1261 [2026-10-20 12:06:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3859 [2026-10-20 12:06:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1470 [2026-10-20 12:06:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7569 [2026-10-20 12:06:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8849 [2026-10-20 12:06:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8815 [2026-10-20 12:06:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9716 [2026-10-20 12:06:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7593 [2026-10-20 12:06:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6075 [2026-10-20 12:06:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1058 [2026-10-20 12:07:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6183 [2026-10-20 12:07:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7647 [2026-10-20 12:07:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6919 [2026-10-20 12:07:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2070 [2026-10-20 12:07:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2592 [2026-10-20 12:07:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3269 [2026-10-20 12:07:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5184 [2026-10-20 12:07:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5027 [2026-10-20 12:07:42][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,567 | MFU: 1.8% [2026-10-20 12:07:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2333 [2026-10-20 12:07:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9153 [2026-10-20 12:07:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1207 [2026-10-20 12:07:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8844 [2026-10-20 12:07:47][INFO][training.pretrain_engine] [Ph2|Step 437] loss=6.1287 | ppl=458.83 | aux=0.02512 | lr=2.81e-04 | gnorm=1.227 | tok/s=29,178 | MFU=7.0% | step=143750ms | tokens=0.466B | mem=4.2/32.7GB | cost=$229.63/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 06:30 (138.4h) | Phase3: SKIPPED | SFT: Oct 26 11:15 (143.1h) | DPO (Final): Oct 26 11:49 (143.7h) [2026-10-20 12:07:47][INFO][training.pretrain_engine] Starting global step 438 (accumulating 32 micro-batches)... [2026-10-20 12:07:47][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 12:08:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1708 [2026-10-20 12:08:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2751 [2026-10-20 12:08:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1999 [2026-10-20 12:08:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2471 [2026-10-20 12:08:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5358 [2026-10-20 12:08:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3174 [2026-10-20 12:08:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7072 [2026-10-20 12:08:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5899 [2026-10-20 12:08:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1832 [2026-10-20 12:08:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4761 [2026-10-20 12:08:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7828 [2026-10-20 12:08:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4401 [2026-10-20 12:08:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4869 [2026-10-20 12:08:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8851 [2026-10-20 12:08:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0716 [2026-10-20 12:08:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3467 [2026-10-20 12:09:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7739 [2026-10-20 12:09:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0480 [2026-10-20 12:09:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1662 [2026-10-20 12:09:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8068 [2026-10-20 12:09:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8536 [2026-10-20 12:09:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4140 [2026-10-20 12:09:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9952 [2026-10-20 12:09:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4418 [2026-10-20 12:09:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8373 [2026-10-20 12:09:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0171 [2026-10-20 12:09:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1110 [2026-10-20 12:09:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1357 [2026-10-20 12:10:06][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,582 | MFU: 1.8% [2026-10-20 12:10:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9445 [2026-10-20 12:10:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3866 [2026-10-20 12:10:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6280 [2026-10-20 12:10:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2828 [2026-10-20 12:10:11][INFO][training.pretrain_engine] [Ph2|Step 438] loss=6.0644 | ppl=430.28 | aux=0.02510 | lr=2.82e-04 | gnorm=0.848 | tok/s=29,177 | MFU=7.0% | step=143754ms | tokens=0.467B | mem=4.2/32.7GB | cost=$230.16/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 06:32 (138.4h) | Phase3: SKIPPED | SFT: Oct 26 11:17 (143.1h) | DPO (Final): Oct 26 11:51 (143.7h) [2026-10-20 12:10:11][INFO][training.pretrain_engine] Starting global step 439 (accumulating 32 micro-batches)... [2026-10-20 12:10:11][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 12:10:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1599 [2026-10-20 12:10:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6182 [2026-10-20 12:10:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3574 [2026-10-20 12:10:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3905 [2026-10-20 12:10:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2740 [2026-10-20 12:10:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9374 [2026-10-20 12:10:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1703 [2026-10-20 12:10:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5583 [2026-10-20 12:11:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8740 [2026-10-20 12:11:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1745 [2026-10-20 12:11:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7821 [2026-10-20 12:11:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3877 [2026-10-20 12:11:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6811 [2026-10-20 12:11:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0111 [2026-10-20 12:11:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2647 [2026-10-20 12:11:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8496 [2026-10-20 12:11:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9352 [2026-10-20 12:11:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1333 [2026-10-20 12:11:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7096 [2026-10-20 12:11:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2764 [2026-10-20 12:11:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7952 [2026-10-20 12:11:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0403 [2026-10-20 12:11:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1359 [2026-10-20 12:11:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8316 [2026-10-20 12:12:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9545 [2026-10-20 12:12:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9262 [2026-10-20 12:12:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1481 [2026-10-20 12:12:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6965 [2026-10-20 12:12:30][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,561 | MFU: 1.8% [2026-10-20 12:12:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9631 [2026-10-20 12:12:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3587 [2026-10-20 12:12:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9778 [2026-10-20 12:12:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8714 [2026-10-20 12:12:35][INFO][training.pretrain_engine] [Ph2|Step 439] loss=6.0546 | ppl=426.08 | aux=0.02511 | lr=2.83e-04 | gnorm=0.914 | tok/s=29,177 | MFU=7.0% | step=143752ms | tokens=0.468B | mem=4.2/32.7GB | cost=$230.68/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 06:33 (138.4h) | Phase3: SKIPPED | SFT: Oct 26 11:19 (143.1h) | DPO (Final): Oct 26 11:53 (143.7h) [2026-10-20 12:12:35][INFO][training.pretrain_engine] Starting global step 440 (accumulating 32 micro-batches)... [2026-10-20 12:12:35][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 12:12:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5793 [2026-10-20 12:12:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3906 [2026-10-20 12:12:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5086 [2026-10-20 12:12:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3660 [2026-10-20 12:13:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7351 [2026-10-20 12:13:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2601 [2026-10-20 12:13:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1707 [2026-10-20 12:13:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1920 [2026-10-20 12:13:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3823 [2026-10-20 12:13:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0438 [2026-10-20 12:13:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0913 [2026-10-20 12:13:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8342 [2026-10-20 12:13:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2818 [2026-10-20 12:13:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6085 [2026-10-20 12:13:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0699 [2026-10-20 12:13:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2867 [2026-10-20 12:14:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6992 [2026-10-20 12:14:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4928 [2026-10-20 12:14:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9885 [2026-10-20 12:14:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5943 [2026-10-20 12:14:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6166 [2026-10-20 12:14:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5734 [2026-10-20 12:14:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1163 [2026-10-20 12:14:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3941 [2026-10-20 12:14:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6380 [2026-10-20 12:14:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1352 [2026-10-20 12:14:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7612 [2026-10-20 12:14:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2306 [2026-10-20 12:14:55][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,504 | MFU: 1.8% [2026-10-20 12:14:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4170 [2026-10-20 12:14:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2200 [2026-10-20 12:14:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5541 [2026-10-20 12:14:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2400 [2026-10-20 12:14:59][INFO][training.pretrain_engine] [Ph2|Step 440] loss=6.1017 | ppl=446.62 | aux=0.02510 | lr=2.83e-04 | gnorm=1.039 | tok/s=29,176 | MFU=7.0% | step=143759ms | tokens=0.469B | mem=4.2/32.7GB | cost=$231.21/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 06:35 (138.3h) | Phase3: SKIPPED | SFT: Oct 26 11:21 (143.1h) | DPO (Final): Oct 26 11:55 (143.7h) [2026-10-20 12:14:59][INFO][training.pretrain_engine] Starting global step 441 (accumulating 32 micro-batches)... [2026-10-20 12:14:59][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 12:15:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1406 [2026-10-20 12:15:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0237 [2026-10-20 12:15:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1601 [2026-10-20 12:15:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3819 [2026-10-20 12:15:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6602 [2026-10-20 12:15:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6766 [2026-10-20 12:15:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6883 [2026-10-20 12:15:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6534 [2026-10-20 12:15:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7568 [2026-10-20 12:15:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8191 [2026-10-20 12:15:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5003 [2026-10-20 12:15:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0404 [2026-10-20 12:16:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3888 [2026-10-20 12:16:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.7380 [2026-10-20 12:16:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4881 [2026-10-20 12:16:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6183 [2026-10-20 12:16:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9194 [2026-10-20 12:16:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6226 [2026-10-20 12:16:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9568 [2026-10-20 12:16:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6300 [2026-10-20 12:16:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0330 [2026-10-20 12:16:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1770 [2026-10-20 12:16:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0977 [2026-10-20 12:16:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0948 [2026-10-20 12:17:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2432 [2026-10-20 12:17:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8031 [2026-10-20 12:17:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1410 [2026-10-20 12:17:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5295 [2026-10-20 12:17:18][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,542 | MFU: 1.8% [2026-10-20 12:17:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3029 [2026-10-20 12:17:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4906 [2026-10-20 12:17:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1064 [2026-10-20 12:17:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6211 [2026-10-20 12:17:23][INFO][training.pretrain_engine] [Ph2|Step 441] loss=6.0602 | ppl=428.44 | aux=0.02513 | lr=2.84e-04 | gnorm=1.156 | tok/s=29,176 | MFU=7.0% | step=143757ms | tokens=0.470B | mem=4.2/32.7GB | cost=$231.74/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 06:37 (138.3h) | Phase3: SKIPPED | SFT: Oct 26 11:23 (143.1h) | DPO (Final): Oct 26 11:57 (143.7h) [2026-10-20 12:17:26][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step441 | Tokens: 0.470B | Spent: $231.75 | Remaining: $168.25 [2026-10-20 12:17:26][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step441 [2026-10-20 12:17:26][INFO][training.pretrain_engine] Starting global step 442 (accumulating 32 micro-batches)... [2026-10-20 12:17:26][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 12:17:26][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 12:17:26][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 12:17:26][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 12:17:26][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step441 to Phase-Technologies/forge-500m-pretrain (path: phase2_step441)... [2026-10-20 12:17:26][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 12:17:36][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 12:17:36][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step441 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 12:17:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7594 [2026-10-20 12:17:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1743 [2026-10-20 12:17:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4130 [2026-10-20 12:17:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7713 [2026-10-20 12:17:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4569 [2026-10-20 12:17:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2541 [2026-10-20 12:17:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2358 [2026-10-20 12:18:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2211 [2026-10-20 12:18:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2439 [2026-10-20 12:18:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0712 [2026-10-20 12:18:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8379 [2026-10-20 12:18:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1920 [2026-10-20 12:18:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2212 [2026-10-20 12:18:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3808 [2026-10-20 12:18:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3444 [2026-10-20 12:18:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2924 [2026-10-20 12:18:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3413 [2026-10-20 12:18:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1339 [2026-10-20 12:18:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0782 [2026-10-20 12:18:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3351 [2026-10-20 12:19:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4140 [2026-10-20 12:19:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1769 [2026-10-20 12:19:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9321 [2026-10-20 12:19:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2122 [2026-10-20 12:19:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6734 [2026-10-20 12:19:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6500 [2026-10-20 12:19:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2147 [2026-10-20 12:19:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7639 [2026-10-20 12:19:45][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,518 | MFU: 1.8% [2026-10-20 12:19:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0798 [2026-10-20 12:19:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8908 [2026-10-20 12:19:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0892 [2026-10-20 12:19:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0357 [2026-10-20 12:19:50][INFO][training.pretrain_engine] [Ph2|Step 442] loss=6.0937 | ppl=443.06 | aux=0.02512 | lr=2.84e-04 | gnorm=1.352 | tok/s=29,176 | MFU=7.0% | step=143760ms | tokens=0.471B | mem=4.2/32.7GB | cost=$232.28/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 06:39 (138.3h) | Phase3: SKIPPED | SFT: Oct 26 11:25 (143.1h) | DPO (Final): Oct 26 11:59 (143.7h) [2026-10-20 12:19:50][INFO][training.pretrain_engine] Starting global step 443 (accumulating 32 micro-batches)... [2026-10-20 12:19:50][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 12:20:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9132 [2026-10-20 12:20:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0324 [2026-10-20 12:20:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1364 [2026-10-20 12:20:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8910 [2026-10-20 12:20:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1769 [2026-10-20 12:20:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5623 [2026-10-20 12:20:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1475 [2026-10-20 12:20:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3864 [2026-10-20 12:20:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2317 [2026-10-20 12:20:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4916 [2026-10-20 12:20:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3858 [2026-10-20 12:20:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9537 [2026-10-20 12:21:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8321 [2026-10-20 12:21:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5594 [2026-10-20 12:21:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9620 [2026-10-20 12:21:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1850 [2026-10-20 12:21:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9382 [2026-10-20 12:21:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0628 [2026-10-20 12:21:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0906 [2026-10-20 12:21:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5209 [2026-10-20 12:21:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3572 [2026-10-20 12:21:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8382 [2026-10-20 12:21:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1448 [2026-10-20 12:21:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3697 [2026-10-20 12:21:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7954 [2026-10-20 12:21:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9051 [2026-10-20 12:21:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1513 [2026-10-20 12:21:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1217 [2026-10-20 12:22:09][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,527 | MFU: 1.8% [2026-10-20 12:22:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4885 [2026-10-20 12:22:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4896 [2026-10-20 12:22:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4035 [2026-10-20 12:22:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3249 [2026-10-20 12:22:14][INFO][training.pretrain_engine] [Ph2|Step 443] loss=6.0640 | ppl=430.09 | aux=0.02510 | lr=2.85e-04 | gnorm=0.719 | tok/s=29,176 | MFU=7.0% | step=143758ms | tokens=0.472B | mem=4.2/32.7GB | cost=$232.80/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 06:41 (138.3h) | Phase3: SKIPPED | SFT: Oct 26 11:26 (143.1h) | DPO (Final): Oct 26 12:01 (143.7h) [2026-10-20 12:22:14][INFO][training.pretrain_engine] Starting global step 444 (accumulating 32 micro-batches)... [2026-10-20 12:22:14][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 12:22:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4510 [2026-10-20 12:22:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7428 [2026-10-20 12:22:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0755 [2026-10-20 12:22:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9838 [2026-10-20 12:22:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3837 [2026-10-20 12:22:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8132 [2026-10-20 12:22:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=7.2381 [2026-10-20 12:22:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4227 [2026-10-20 12:23:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4614 [2026-10-20 12:23:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6524 [2026-10-20 12:23:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7653 [2026-10-20 12:23:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6449 [2026-10-20 12:23:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8054 [2026-10-20 12:23:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2472 [2026-10-20 12:23:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6789 [2026-10-20 12:23:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0073 [2026-10-20 12:23:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3185 [2026-10-20 12:23:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9788 [2026-10-20 12:23:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3862 [2026-10-20 12:23:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6268 [2026-10-20 12:24:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4424 [2026-10-20 12:24:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3881 [2026-10-20 12:24:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5749 [2026-10-20 12:24:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9542 [2026-10-20 12:24:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3579 [2026-10-20 12:24:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3372 [2026-10-20 12:24:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4614 [2026-10-20 12:24:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6671 [2026-10-20 12:24:34][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,507 | MFU: 1.8% [2026-10-20 12:24:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1055 [2026-10-20 12:24:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0757 [2026-10-20 12:24:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7038 [2026-10-20 12:24:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0521 [2026-10-20 12:24:38][INFO][training.pretrain_engine] [Ph2|Step 444] loss=6.2013 | ppl=493.41 | aux=0.02512 | lr=2.85e-04 | gnorm=1.031 | tok/s=29,175 | MFU=7.0% | step=143764ms | tokens=0.473B | mem=4.2/32.7GB | cost=$233.33/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 06:43 (138.3h) | Phase3: SKIPPED | SFT: Oct 26 11:29 (143.1h) | DPO (Final): Oct 26 12:03 (143.6h) [2026-10-20 12:24:38][INFO][training.pretrain_engine] Starting global step 445 (accumulating 32 micro-batches)... [2026-10-20 12:24:38][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 12:24:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4945 [2026-10-20 12:24:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4818 [2026-10-20 12:24:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8591 [2026-10-20 12:24:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5435 [2026-10-20 12:25:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7576 [2026-10-20 12:25:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5129 [2026-10-20 12:25:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7099 [2026-10-20 12:25:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6936 [2026-10-20 12:25:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9726 [2026-10-20 12:25:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9199 [2026-10-20 12:25:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0165 [2026-10-20 12:25:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4316 [2026-10-20 12:25:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1437 [2026-10-20 12:25:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4178 [2026-10-20 12:25:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7198 [2026-10-20 12:25:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5469 [2026-10-20 12:26:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6422 [2026-10-20 12:26:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0644 [2026-10-20 12:26:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4886 [2026-10-20 12:26:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0688 [2026-10-20 12:26:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7867 [2026-10-20 12:26:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9421 [2026-10-20 12:26:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8155 [2026-10-20 12:26:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0224 [2026-10-20 12:26:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7164 [2026-10-20 12:26:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6962 [2026-10-20 12:26:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2358 [2026-10-20 12:26:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2471 [2026-10-20 12:26:57][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,535 | MFU: 1.8% [2026-10-20 12:27:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7890 [2026-10-20 12:27:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4565 [2026-10-20 12:27:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7303 [2026-10-20 12:27:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0802 [2026-10-20 12:27:02][INFO][training.pretrain_engine] [Ph2|Step 445] loss=6.0601 | ppl=428.42 | aux=0.02512 | lr=2.86e-04 | gnorm=1.000 | tok/s=29,177 | MFU=7.0% | step=143753ms | tokens=0.474B | mem=4.2/32.7GB | cost=$233.85/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 06:44 (138.3h) | Phase3: SKIPPED | SFT: Oct 26 11:30 (143.1h) | DPO (Final): Oct 26 12:04 (143.6h) [2026-10-20 12:27:02][INFO][training.pretrain_engine] Starting global step 446 (accumulating 32 micro-batches)... [2026-10-20 12:27:02][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 12:27:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9538 [2026-10-20 12:27:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1627 [2026-10-20 12:27:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3174 [2026-10-20 12:27:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1511 [2026-10-20 12:27:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7693 [2026-10-20 12:27:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0336 [2026-10-20 12:27:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7959 [2026-10-20 12:27:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9900 [2026-10-20 12:27:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9927 [2026-10-20 12:27:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0210 [2026-10-20 12:27:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2517 [2026-10-20 12:27:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0043 [2026-10-20 12:28:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2400 [2026-10-20 12:28:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7133 [2026-10-20 12:28:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5053 [2026-10-20 12:28:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4950 [2026-10-20 12:28:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5328 [2026-10-20 12:28:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3321 [2026-10-20 12:28:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8957 [2026-10-20 12:28:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2251 [2026-10-20 12:28:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0121 [2026-10-20 12:28:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2400 [2026-10-20 12:28:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9186 [2026-10-20 12:28:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9886 [2026-10-20 12:29:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5522 [2026-10-20 12:29:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8711 [2026-10-20 12:29:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1003 [2026-10-20 12:29:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7958 [2026-10-20 12:29:20][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,564 | MFU: 1.8% [2026-10-20 12:29:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6630 [2026-10-20 12:29:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5464 [2026-10-20 12:29:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7806 [2026-10-20 12:29:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0374 [2026-10-20 12:29:25][INFO][training.pretrain_engine] [Ph2|Step 446] loss=5.9128 | ppl=369.75 | aux=0.02510 | lr=2.87e-04 | gnorm=1.031 | tok/s=29,178 | MFU=7.0% | step=143750ms | tokens=0.475B | mem=4.2/32.7GB | cost=$234.38/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 06:46 (138.3h) | Phase3: SKIPPED | SFT: Oct 26 11:31 (143.0h) | DPO (Final): Oct 26 12:06 (143.6h) [2026-10-20 12:29:25][INFO][training.pretrain_engine] Starting global step 447 (accumulating 32 micro-batches)... [2026-10-20 12:29:25][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 12:29:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1780 [2026-10-20 12:29:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2994 [2026-10-20 12:29:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7404 [2026-10-20 12:29:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8747 [2026-10-20 12:30:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1688 [2026-10-20 12:30:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0858 [2026-10-20 12:30:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0546 [2026-10-20 12:30:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0745 [2026-10-20 12:30:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6146 [2026-10-20 12:30:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2467 [2026-10-20 12:30:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9639 [2026-10-20 12:30:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0981 [2026-10-20 12:30:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4822 [2026-10-20 12:30:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0903 [2026-10-20 12:30:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8177 [2026-10-20 12:30:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3378 [2026-10-20 12:30:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1555 [2026-10-20 12:30:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1808 [2026-10-20 12:30:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3344 [2026-10-20 12:30:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5783 [2026-10-20 12:31:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8762 [2026-10-20 12:31:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7278 [2026-10-20 12:31:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9739 [2026-10-20 12:31:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8272 [2026-10-20 12:31:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9720 [2026-10-20 12:31:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8757 [2026-10-20 12:31:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1276 [2026-10-20 12:31:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8804 [2026-10-20 12:31:44][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,572 | MFU: 1.8% [2026-10-20 12:31:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9139 [2026-10-20 12:31:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8110 [2026-10-20 12:31:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9677 [2026-10-20 12:31:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1291 [2026-10-20 12:31:49][INFO][training.pretrain_engine] [Ph2|Step 447] loss=6.1225 | ppl=456.01 | aux=0.02509 | lr=2.87e-04 | gnorm=1.055 | tok/s=29,177 | MFU=7.0% | step=143753ms | tokens=0.476B | mem=4.2/32.7GB | cost=$234.90/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 06:48 (138.3h) | Phase3: SKIPPED | SFT: Oct 26 11:33 (143.0h) | DPO (Final): Oct 26 12:08 (143.6h) [2026-10-20 12:31:49][INFO][training.pretrain_engine] Starting global step 448 (accumulating 32 micro-batches)... [2026-10-20 12:31:49][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 12:32:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9342 [2026-10-20 12:32:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3432 [2026-10-20 12:32:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5664 [2026-10-20 12:32:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6222 [2026-10-20 12:32:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0137 [2026-10-20 12:32:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1305 [2026-10-20 12:32:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4462 [2026-10-20 12:32:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7090 [2026-10-20 12:32:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2111 [2026-10-20 12:32:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1801 [2026-10-20 12:32:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9911 [2026-10-20 12:32:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9807 [2026-10-20 12:33:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6238 [2026-10-20 12:33:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0361 [2026-10-20 12:33:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0660 [2026-10-20 12:33:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7480 [2026-10-20 12:33:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3508 [2026-10-20 12:33:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2682 [2026-10-20 12:33:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6582 [2026-10-20 12:33:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8111 [2026-10-20 12:33:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3685 [2026-10-20 12:33:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8242 [2026-10-20 12:33:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9357 [2026-10-20 12:33:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6448 [2026-10-20 12:33:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5257 [2026-10-20 12:33:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8022 [2026-10-20 12:33:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3714 [2026-10-20 12:33:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8127 [2026-10-20 12:34:08][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,532 | MFU: 1.8% [2026-10-20 12:34:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9949 [2026-10-20 12:34:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0813 [2026-10-20 12:34:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5411 [2026-10-20 12:34:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8984 [2026-10-20 12:34:13][INFO][training.pretrain_engine] [Ph2|Step 448] loss=6.0412 | ppl=420.39 | aux=0.02510 | lr=2.88e-04 | gnorm=1.195 | tok/s=29,177 | MFU=7.0% | step=143755ms | tokens=0.477B | mem=4.2/32.7GB | cost=$235.43/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 06:50 (138.3h) | Phase3: SKIPPED | SFT: Oct 26 11:35 (143.0h) | DPO (Final): Oct 26 12:10 (143.6h) [2026-10-20 12:34:13][INFO][training.pretrain_engine] Starting global step 449 (accumulating 32 micro-batches)... [2026-10-20 12:34:13][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 12:34:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9829 [2026-10-20 12:34:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5235 [2026-10-20 12:34:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1624 [2026-10-20 12:34:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0459 [2026-10-20 12:34:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7234 [2026-10-20 12:34:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4568 [2026-10-20 12:34:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5988 [2026-10-20 12:34:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6872 [2026-10-20 12:35:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4249 [2026-10-20 12:35:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6296 [2026-10-20 12:35:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9731 [2026-10-20 12:35:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9992 [2026-10-20 12:35:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5016 [2026-10-20 12:35:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1874 [2026-10-20 12:35:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9581 [2026-10-20 12:35:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2833 [2026-10-20 12:35:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1245 [2026-10-20 12:35:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0058 [2026-10-20 12:35:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6535 [2026-10-20 12:35:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.1624 [2026-10-20 12:36:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8674 [2026-10-20 12:36:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8566 [2026-10-20 12:36:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5196 [2026-10-20 12:36:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8912 [2026-10-20 12:36:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7241 [2026-10-20 12:36:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8277 [2026-10-20 12:36:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2503 [2026-10-20 12:36:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7303 [2026-10-20 12:36:33][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,523 | MFU: 1.8% [2026-10-20 12:36:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3895 [2026-10-20 12:36:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5495 [2026-10-20 12:36:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8383 [2026-10-20 12:36:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1499 [2026-10-20 12:36:37][INFO][training.pretrain_engine] [Ph2|Step 449] loss=6.1544 | ppl=470.77 | aux=0.02512 | lr=2.88e-04 | gnorm=1.148 | tok/s=29,176 | MFU=7.0% | step=143758ms | tokens=0.478B | mem=4.2/32.7GB | cost=$235.96/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 06:52 (138.3h) | Phase3: SKIPPED | SFT: Oct 26 11:37 (143.0h) | DPO (Final): Oct 26 12:12 (143.6h) [2026-10-20 12:36:37][INFO][training.pretrain_engine] Starting global step 450 (accumulating 32 micro-batches)... [2026-10-20 12:36:37][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 12:36:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3752 [2026-10-20 12:36:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2171 [2026-10-20 12:36:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0920 [2026-10-20 12:36:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1489 [2026-10-20 12:37:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7615 [2026-10-20 12:37:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3784 [2026-10-20 12:37:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2801 [2026-10-20 12:37:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8572 [2026-10-20 12:37:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9126 [2026-10-20 12:37:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0998 [2026-10-20 12:37:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6827 [2026-10-20 12:37:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1984 [2026-10-20 12:37:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8825 [2026-10-20 12:37:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9828 [2026-10-20 12:37:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3254 [2026-10-20 12:37:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6227 [2026-10-20 12:38:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1658 [2026-10-20 12:38:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7488 [2026-10-20 12:38:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5541 [2026-10-20 12:38:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4004 [2026-10-20 12:38:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0445 [2026-10-20 12:38:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1252 [2026-10-20 12:38:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8691 [2026-10-20 12:38:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4211 [2026-10-20 12:38:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8966 [2026-10-20 12:38:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8592 [2026-10-20 12:38:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7038 [2026-10-20 12:38:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0583 [2026-10-20 12:38:56][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,555 | MFU: 1.8% [2026-10-20 12:39:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9480 [2026-10-20 12:39:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9504 [2026-10-20 12:39:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7142 [2026-10-20 12:39:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1293 [2026-10-20 12:39:01][INFO][training.pretrain_engine] [Ph2|Step 450] loss=6.0708 | ppl=433.03 | aux=0.02510 | lr=2.89e-04 | gnorm=0.906 | tok/s=29,177 | MFU=7.0% | step=143755ms | tokens=0.479B | mem=4.2/32.7GB | cost=$236.48/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 06:53 (138.2h) | Phase3: SKIPPED | SFT: Oct 26 11:39 (143.0h) | DPO (Final): Oct 26 12:13 (143.6h) [2026-10-20 12:39:04][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step450 | Tokens: 0.479B | Spent: $236.49 | Remaining: $163.51 [2026-10-20 12:39:04][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step450 [2026-10-20 12:39:04][INFO][training.pretrain_engine] Starting global step 451 (accumulating 32 micro-batches)... [2026-10-20 12:39:04][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 12:39:04][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 12:39:04][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 12:39:04][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 12:39:04][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step450 to Phase-Technologies/forge-500m-pretrain (path: phase2_step450)... [2026-10-20 12:39:04][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 12:39:16][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 12:39:16][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step450 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 12:39:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2658 [2026-10-20 12:39:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5080 [2026-10-20 12:39:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3213 [2026-10-20 12:39:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4805 [2026-10-20 12:39:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7373 [2026-10-20 12:39:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8317 [2026-10-20 12:39:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3895 [2026-10-20 12:39:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1944 [2026-10-20 12:39:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1585 [2026-10-20 12:39:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8648 [2026-10-20 12:39:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5896 [2026-10-20 12:39:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2563 [2026-10-20 12:40:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9275 [2026-10-20 12:40:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6405 [2026-10-20 12:40:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3896 [2026-10-20 12:40:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1781 [2026-10-20 12:40:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2015 [2026-10-20 12:40:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2366 [2026-10-20 12:40:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8481 [2026-10-20 12:40:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0692 [2026-10-20 12:40:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7190 [2026-10-20 12:40:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8215 [2026-10-20 12:40:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2749 [2026-10-20 12:40:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6969 [2026-10-20 12:41:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3427 [2026-10-20 12:41:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0572 [2026-10-20 12:41:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0703 [2026-10-20 12:41:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9019 [2026-10-20 12:41:23][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,567 | MFU: 1.8% [2026-10-20 12:41:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3304 [2026-10-20 12:41:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5963 [2026-10-20 12:41:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9026 [2026-10-20 12:41:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2898 [2026-10-20 12:41:27][INFO][training.pretrain_engine] [Ph2|Step 451] loss=6.0363 | ppl=418.36 | aux=0.02510 | lr=2.89e-04 | gnorm=0.973 | tok/s=29,179 | MFU=7.0% | step=143744ms | tokens=0.480B | mem=4.2/32.7GB | cost=$237.02/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 06:54 (138.2h) | Phase3: SKIPPED | SFT: Oct 26 11:40 (143.0h) | DPO (Final): Oct 26 12:14 (143.6h) [2026-10-20 12:41:27][INFO][training.pretrain_engine] Starting global step 452 (accumulating 32 micro-batches)... [2026-10-20 12:41:27][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 12:41:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9999 [2026-10-20 12:41:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0620 [2026-10-20 12:41:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4315 [2026-10-20 12:41:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1261 [2026-10-20 12:42:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8578 [2026-10-20 12:42:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6553 [2026-10-20 12:42:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0764 [2026-10-20 12:42:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4676 [2026-10-20 12:42:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9564 [2026-10-20 12:42:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8506 [2026-10-20 12:42:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7502 [2026-10-20 12:42:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7210 [2026-10-20 12:42:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8283 [2026-10-20 12:42:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5266 [2026-10-20 12:42:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8430 [2026-10-20 12:42:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8850 [2026-10-20 12:42:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2395 [2026-10-20 12:42:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0258 [2026-10-20 12:42:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2525 [2026-10-20 12:42:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1902 [2026-10-20 12:43:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7753 [2026-10-20 12:43:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3852 [2026-10-20 12:43:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9321 [2026-10-20 12:43:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5155 [2026-10-20 12:43:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0941 [2026-10-20 12:43:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9174 [2026-10-20 12:43:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1831 [2026-10-20 12:43:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3518 [2026-10-20 12:43:46][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,564 | MFU: 1.8% [2026-10-20 12:43:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1682 [2026-10-20 12:43:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1143 [2026-10-20 12:43:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7938 [2026-10-20 12:43:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6566 [2026-10-20 12:43:50][INFO][training.pretrain_engine] [Ph2|Step 452] loss=5.9834 | ppl=396.80 | aux=0.02510 | lr=2.90e-04 | gnorm=1.195 | tok/s=29,181 | MFU=7.0% | step=143736ms | tokens=0.481B | mem=4.2/32.7GB | cost=$237.54/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 06:56 (138.2h) | Phase3: SKIPPED | SFT: Oct 26 11:41 (143.0h) | DPO (Final): Oct 26 12:16 (143.5h) [2026-10-20 12:43:50][INFO][training.pretrain_engine] Starting global step 453 (accumulating 32 micro-batches)... [2026-10-20 12:43:50][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 12:44:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0264 [2026-10-20 12:44:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3707 [2026-10-20 12:44:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1898 [2026-10-20 12:44:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7088 [2026-10-20 12:44:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2083 [2026-10-20 12:44:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6471 [2026-10-20 12:44:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6423 [2026-10-20 12:44:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6262 [2026-10-20 12:44:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9813 [2026-10-20 12:44:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6568 [2026-10-20 12:44:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9326 [2026-10-20 12:44:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0880 [2026-10-20 12:45:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3279 [2026-10-20 12:45:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9084 [2026-10-20 12:45:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9654 [2026-10-20 12:45:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2530 [2026-10-20 12:45:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4081 [2026-10-20 12:45:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4106 [2026-10-20 12:45:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5277 [2026-10-20 12:45:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3736 [2026-10-20 12:45:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7971 [2026-10-20 12:45:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0751 [2026-10-20 12:45:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1090 [2026-10-20 12:45:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2189 [2026-10-20 12:45:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1556 [2026-10-20 12:45:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8832 [2026-10-20 12:45:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6177 [2026-10-20 12:45:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8935 [2026-10-20 12:46:10][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,521 | MFU: 1.8% [2026-10-20 12:46:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0571 [2026-10-20 12:46:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7666 [2026-10-20 12:46:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8237 [2026-10-20 12:46:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9989 [2026-10-20 12:46:14][INFO][training.pretrain_engine] [Ph2|Step 453] loss=6.0088 | ppl=406.99 | aux=0.02510 | lr=2.90e-04 | gnorm=0.684 | tok/s=29,180 | MFU=7.0% | step=143738ms | tokens=0.482B | mem=4.2/32.7GB | cost=$238.07/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 06:58 (138.2h) | Phase3: SKIPPED | SFT: Oct 26 11:43 (143.0h) | DPO (Final): Oct 26 12:18 (143.5h) [2026-10-20 12:46:14][INFO][training.pretrain_engine] Starting global step 454 (accumulating 32 micro-batches)... [2026-10-20 12:46:14][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 12:46:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6200 [2026-10-20 12:46:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4902 [2026-10-20 12:46:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2956 [2026-10-20 12:46:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1666 [2026-10-20 12:46:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5156 [2026-10-20 12:46:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1040 [2026-10-20 12:46:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7759 [2026-10-20 12:46:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8633 [2026-10-20 12:47:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0587 [2026-10-20 12:47:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8716 [2026-10-20 12:47:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3090 [2026-10-20 12:47:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8367 [2026-10-20 12:47:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7189 [2026-10-20 12:47:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6510 [2026-10-20 12:47:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7252 [2026-10-20 12:47:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7551 [2026-10-20 12:47:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6135 [2026-10-20 12:47:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0667 [2026-10-20 12:47:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0843 [2026-10-20 12:47:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8789 [2026-10-20 12:48:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3832 [2026-10-20 12:48:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7079 [2026-10-20 12:48:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9246 [2026-10-20 12:48:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1858 [2026-10-20 12:48:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0608 [2026-10-20 12:48:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8693 [2026-10-20 12:48:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1602 [2026-10-20 12:48:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0537 [2026-10-20 12:48:33][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,540 | MFU: 1.8% [2026-10-20 12:48:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2357 [2026-10-20 12:48:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9012 [2026-10-20 12:48:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9190 [2026-10-20 12:48:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9332 [2026-10-20 12:48:38][INFO][training.pretrain_engine] [Ph2|Step 454] loss=5.8857 | ppl=359.85 | aux=0.02512 | lr=2.91e-04 | gnorm=1.070 | tok/s=29,180 | MFU=7.0% | step=143739ms | tokens=0.483B | mem=4.2/32.7GB | cost=$238.59/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 07:00 (138.2h) | Phase3: SKIPPED | SFT: Oct 26 11:45 (142.9h) | DPO (Final): Oct 26 12:19 (143.5h) [2026-10-20 12:48:38][INFO][training.pretrain_engine] Starting global step 455 (accumulating 32 micro-batches)... [2026-10-20 12:48:38][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 12:48:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7249 [2026-10-20 12:48:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5512 [2026-10-20 12:48:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7235 [2026-10-20 12:48:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1502 [2026-10-20 12:49:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8095 [2026-10-20 12:49:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2873 [2026-10-20 12:49:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7593 [2026-10-20 12:49:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1668 [2026-10-20 12:49:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0368 [2026-10-20 12:49:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0065 [2026-10-20 12:49:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0212 [2026-10-20 12:49:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9253 [2026-10-20 12:49:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6229 [2026-10-20 12:49:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3995 [2026-10-20 12:49:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6834 [2026-10-20 12:49:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5731 [2026-10-20 12:50:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3221 [2026-10-20 12:50:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1498 [2026-10-20 12:50:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4402 [2026-10-20 12:50:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0510 [2026-10-20 12:50:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1240 [2026-10-20 12:50:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0012 [2026-10-20 12:50:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1911 [2026-10-20 12:50:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1680 [2026-10-20 12:50:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0528 [2026-10-20 12:50:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1520 [2026-10-20 12:50:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9949 [2026-10-20 12:50:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0431 [2026-10-20 12:50:57][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,545 | MFU: 1.8% [2026-10-20 12:51:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0363 [2026-10-20 12:51:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6154 [2026-10-20 12:51:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7580 [2026-10-20 12:51:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0838 [2026-10-20 12:51:01][INFO][training.pretrain_engine] [Ph2|Step 455] loss=5.9818 | ppl=396.15 | aux=0.02511 | lr=2.92e-04 | gnorm=1.234 | tok/s=29,181 | MFU=7.0% | step=143736ms | tokens=0.484B | mem=4.2/32.7GB | cost=$239.12/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 07:01 (138.2h) | Phase3: SKIPPED | SFT: Oct 26 11:47 (142.9h) | DPO (Final): Oct 26 12:21 (143.5h) [2026-10-20 12:51:01][INFO][training.pretrain_engine] Starting global step 456 (accumulating 32 micro-batches)... [2026-10-20 12:51:01][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 12:51:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1630 [2026-10-20 12:51:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9873 [2026-10-20 12:51:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0808 [2026-10-20 12:51:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0860 [2026-10-20 12:51:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9634 [2026-10-20 12:51:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9369 [2026-10-20 12:51:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0809 [2026-10-20 12:51:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4168 [2026-10-20 12:51:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9060 [2026-10-20 12:51:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5504 [2026-10-20 12:51:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7690 [2026-10-20 12:51:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6859 [2026-10-20 12:52:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0378 [2026-10-20 12:52:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9483 [2026-10-20 12:52:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0059 [2026-10-20 12:52:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2869 [2026-10-20 12:52:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0976 [2026-10-20 12:52:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7633 [2026-10-20 12:52:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7164 [2026-10-20 12:52:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8600 [2026-10-20 12:52:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6294 [2026-10-20 12:52:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7633 [2026-10-20 12:52:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1584 [2026-10-20 12:52:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8282 [2026-10-20 12:53:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8257 [2026-10-20 12:53:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2283 [2026-10-20 12:53:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0191 [2026-10-20 12:53:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0422 [2026-10-20 12:53:20][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,543 | MFU: 1.8% [2026-10-20 12:53:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9703 [2026-10-20 12:53:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0338 [2026-10-20 12:53:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3433 [2026-10-20 12:53:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9998 [2026-10-20 12:53:25][INFO][training.pretrain_engine] [Ph2|Step 456] loss=6.0217 | ppl=412.28 | aux=0.02509 | lr=2.92e-04 | gnorm=0.957 | tok/s=29,182 | MFU=7.0% | step=143727ms | tokens=0.485B | mem=4.2/32.7GB | cost=$239.64/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 07:02 (138.2h) | Phase3: SKIPPED | SFT: Oct 26 11:48 (142.9h) | DPO (Final): Oct 26 12:22 (143.5h) [2026-10-20 12:53:25][INFO][training.pretrain_engine] Starting global step 457 (accumulating 32 micro-batches)... [2026-10-20 12:53:25][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 12:53:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9602 [2026-10-20 12:53:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2659 [2026-10-20 12:53:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5971 [2026-10-20 12:53:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9396 [2026-10-20 12:54:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8969 [2026-10-20 12:54:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2951 [2026-10-20 12:54:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4855 [2026-10-20 12:54:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4609 [2026-10-20 12:54:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8265 [2026-10-20 12:54:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2660 [2026-10-20 12:54:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9489 [2026-10-20 12:54:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1709 [2026-10-20 12:54:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9671 [2026-10-20 12:54:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0247 [2026-10-20 12:54:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1306 [2026-10-20 12:54:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8547 [2026-10-20 12:54:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9072 [2026-10-20 12:54:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6898 [2026-10-20 12:54:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6755 [2026-10-20 12:54:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6104 [2026-10-20 12:55:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2339 [2026-10-20 12:55:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6796 [2026-10-20 12:55:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6516 [2026-10-20 12:55:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7117 [2026-10-20 12:55:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5244 [2026-10-20 12:55:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1401 [2026-10-20 12:55:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4845 [2026-10-20 12:55:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3855 [2026-10-20 12:55:44][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,558 | MFU: 1.8% [2026-10-20 12:55:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1976 [2026-10-20 12:55:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3521 [2026-10-20 12:55:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5269 [2026-10-20 12:55:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9188 [2026-10-20 12:55:48][INFO][training.pretrain_engine] [Ph2|Step 457] loss=5.9968 | ppl=402.15 | aux=0.02510 | lr=2.93e-04 | gnorm=1.133 | tok/s=29,184 | MFU=7.0% | step=143720ms | tokens=0.487B | mem=4.2/32.7GB | cost=$240.16/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 07:04 (138.1h) | Phase3: SKIPPED | SFT: Oct 26 11:49 (142.9h) | DPO (Final): Oct 26 12:24 (143.5h) [2026-10-20 12:55:48][INFO][training.pretrain_engine] Starting global step 458 (accumulating 32 micro-batches)... [2026-10-20 12:55:48][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 12:56:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7934 [2026-10-20 12:56:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0568 [2026-10-20 12:56:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8720 [2026-10-20 12:56:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3873 [2026-10-20 12:56:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0713 [2026-10-20 12:56:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1139 [2026-10-20 12:56:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2189 [2026-10-20 12:56:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0189 [2026-10-20 12:56:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0083 [2026-10-20 12:56:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1715 [2026-10-20 12:56:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8714 [2026-10-20 12:56:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3652 [2026-10-20 12:56:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9626 [2026-10-20 12:56:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8895 [2026-10-20 12:57:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3539 [2026-10-20 12:57:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0303 [2026-10-20 12:57:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5803 [2026-10-20 12:57:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5464 [2026-10-20 12:57:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4615 [2026-10-20 12:57:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3459 [2026-10-20 12:57:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8416 [2026-10-20 12:57:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.8703 [2026-10-20 12:57:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.0522 [2026-10-20 12:57:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7603 [2026-10-20 12:57:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4246 [2026-10-20 12:57:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9872 [2026-10-20 12:57:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2084 [2026-10-20 12:57:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8927 [2026-10-20 12:58:07][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,559 | MFU: 1.8% [2026-10-20 12:58:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0258 [2026-10-20 12:58:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9380 [2026-10-20 12:58:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8981 [2026-10-20 12:58:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0487 [2026-10-20 12:58:11][INFO][training.pretrain_engine] [Ph2|Step 458] loss=6.0802 | ppl=437.11 | aux=0.02510 | lr=2.93e-04 | gnorm=0.902 | tok/s=29,185 | MFU=7.0% | step=143715ms | tokens=0.488B | mem=4.2/32.7GB | cost=$240.69/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 07:05 (138.1h) | Phase3: SKIPPED | SFT: Oct 26 11:51 (142.9h) | DPO (Final): Oct 26 12:25 (143.5h) [2026-10-20 12:58:11][INFO][training.pretrain_engine] Starting global step 459 (accumulating 32 micro-batches)... [2026-10-20 12:58:11][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 12:58:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2949 [2026-10-20 12:58:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1732 [2026-10-20 12:58:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2577 [2026-10-20 12:58:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9513 [2026-10-20 12:58:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1470 [2026-10-20 12:58:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4449 [2026-10-20 12:58:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5180 [2026-10-20 12:58:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0184 [2026-10-20 12:59:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1792 [2026-10-20 12:59:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4817 [2026-10-20 12:59:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2831 [2026-10-20 12:59:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9384 [2026-10-20 12:59:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5276 [2026-10-20 12:59:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3338 [2026-10-20 12:59:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7614 [2026-10-20 12:59:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5335 [2026-10-20 12:59:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5187 [2026-10-20 12:59:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8840 [2026-10-20 12:59:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8582 [2026-10-20 12:59:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9727 [2026-10-20 12:59:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2435 [2026-10-20 12:59:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6055 [2026-10-20 12:59:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8490 [2026-10-20 12:59:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5097 [2026-10-20 13:00:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4945 [2026-10-20 13:00:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9931 [2026-10-20 13:00:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4830 [2026-10-20 13:00:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5642 [2026-10-20 13:00:30][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,551 | MFU: 1.8% [2026-10-20 13:00:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9849 [2026-10-20 13:00:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3415 [2026-10-20 13:00:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9867 [2026-10-20 13:00:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8764 [2026-10-20 13:00:35][INFO][training.pretrain_engine] [Ph2|Step 459] loss=5.9897 | ppl=399.29 | aux=0.02508 | lr=2.94e-04 | gnorm=0.938 | tok/s=29,186 | MFU=7.0% | step=143710ms | tokens=0.489B | mem=4.2/32.7GB | cost=$241.21/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 07:07 (138.1h) | Phase3: SKIPPED | SFT: Oct 26 11:52 (142.9h) | DPO (Final): Oct 26 12:27 (143.4h) [2026-10-20 13:00:38][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step459 | Tokens: 0.489B | Spent: $241.22 | Remaining: $158.78 [2026-10-20 13:00:38][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step459 [2026-10-20 13:00:38][INFO][training.pretrain_engine] Starting global step 460 (accumulating 32 micro-batches)... [2026-10-20 13:00:38][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 13:00:38][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 13:00:38][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 13:00:38][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 13:00:38][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step459 to Phase-Technologies/forge-500m-pretrain (path: phase2_step459)... [2026-10-20 13:00:39][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 13:00:48][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 13:00:48][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step459 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 13:00:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9586 [2026-10-20 13:00:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1089 [2026-10-20 13:00:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0343 [2026-10-20 13:00:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9813 [2026-10-20 13:01:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5622 [2026-10-20 13:01:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8489 [2026-10-20 13:01:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1763 [2026-10-20 13:01:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9647 [2026-10-20 13:01:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1741 [2026-10-20 13:01:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3155 [2026-10-20 13:01:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4097 [2026-10-20 13:01:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9617 [2026-10-20 13:01:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1950 [2026-10-20 13:01:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6172 [2026-10-20 13:01:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5313 [2026-10-20 13:01:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2192 [2026-10-20 13:02:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9410 [2026-10-20 13:02:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6106 [2026-10-20 13:02:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9603 [2026-10-20 13:02:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0868 [2026-10-20 13:02:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9698 [2026-10-20 13:02:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7199 [2026-10-20 13:02:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5573 [2026-10-20 13:02:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7498 [2026-10-20 13:02:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0581 [2026-10-20 13:02:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9186 [2026-10-20 13:02:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3535 [2026-10-20 13:02:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0142 [2026-10-20 13:02:57][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,571 | MFU: 1.8% [2026-10-20 13:03:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7099 [2026-10-20 13:03:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7590 [2026-10-20 13:03:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.8033 [2026-10-20 13:03:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6605 [2026-10-20 13:03:01][INFO][training.pretrain_engine] [Ph2|Step 460] loss=6.1443 | ppl=466.04 | aux=0.02511 | lr=2.94e-04 | gnorm=0.836 | tok/s=29,187 | MFU=7.0% | step=143707ms | tokens=0.490B | mem=4.2/32.7GB | cost=$241.75/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 07:08 (138.1h) | Phase3: SKIPPED | SFT: Oct 26 11:54 (142.9h) | DPO (Final): Oct 26 12:28 (143.4h) [2026-10-20 13:03:01][INFO][training.pretrain_engine] Starting global step 461 (accumulating 32 micro-batches)... [2026-10-20 13:03:01][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 13:03:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0922 [2026-10-20 13:03:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8257 [2026-10-20 13:03:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6153 [2026-10-20 13:03:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3360 [2026-10-20 13:03:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5771 [2026-10-20 13:03:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8934 [2026-10-20 13:03:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7745 [2026-10-20 13:03:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9075 [2026-10-20 13:03:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1225 [2026-10-20 13:03:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1303 [2026-10-20 13:03:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1587 [2026-10-20 13:03:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0883 [2026-10-20 13:04:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9726 [2026-10-20 13:04:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0651 [2026-10-20 13:04:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0603 [2026-10-20 13:04:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0713 [2026-10-20 13:04:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1904 [2026-10-20 13:04:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7177 [2026-10-20 13:04:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4370 [2026-10-20 13:04:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0352 [2026-10-20 13:04:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9327 [2026-10-20 13:04:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2359 [2026-10-20 13:04:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4614 [2026-10-20 13:04:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7200 [2026-10-20 13:05:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6023 [2026-10-20 13:05:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4436 [2026-10-20 13:05:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9148 [2026-10-20 13:05:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4949 [2026-10-20 13:05:20][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,575 | MFU: 1.8% [2026-10-20 13:05:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1316 [2026-10-20 13:05:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1542 [2026-10-20 13:05:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9065 [2026-10-20 13:05:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4602 [2026-10-20 13:05:24][INFO][training.pretrain_engine] [Ph2|Step 461] loss=5.8488 | ppl=346.80 | aux=0.02508 | lr=2.95e-04 | gnorm=0.797 | tok/s=29,187 | MFU=7.0% | step=143706ms | tokens=0.491B | mem=4.2/32.7GB | cost=$242.27/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 07:10 (138.1h) | Phase3: SKIPPED | SFT: Oct 26 11:56 (142.8h) | DPO (Final): Oct 26 12:30 (143.4h) [2026-10-20 13:05:24][INFO][training.pretrain_engine] Starting global step 462 (accumulating 32 micro-batches)... [2026-10-20 13:05:24][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 13:05:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9775 [2026-10-20 13:05:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0936 [2026-10-20 13:05:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6926 [2026-10-20 13:05:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1782 [2026-10-20 13:06:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0176 [2026-10-20 13:06:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9909 [2026-10-20 13:06:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9447 [2026-10-20 13:06:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9288 [2026-10-20 13:06:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4925 [2026-10-20 13:06:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5551 [2026-10-20 13:06:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3454 [2026-10-20 13:06:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6216 [2026-10-20 13:06:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0864 [2026-10-20 13:06:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8749 [2026-10-20 13:06:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3511 [2026-10-20 13:06:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3546 [2026-10-20 13:06:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2194 [2026-10-20 13:06:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1829 [2026-10-20 13:06:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1377 [2026-10-20 13:06:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6255 [2026-10-20 13:07:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8204 [2026-10-20 13:07:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1914 [2026-10-20 13:07:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1940 [2026-10-20 13:07:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2374 [2026-10-20 13:07:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1143 [2026-10-20 13:07:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7920 [2026-10-20 13:07:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4519 [2026-10-20 13:07:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3296 [2026-10-20 13:07:43][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,565 | MFU: 1.8% [2026-10-20 13:07:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0687 [2026-10-20 13:07:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3162 [2026-10-20 13:07:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2770 [2026-10-20 13:07:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9007 [2026-10-20 13:07:48][INFO][training.pretrain_engine] [Ph2|Step 462] loss=6.0380 | ppl=419.05 | aux=0.02510 | lr=2.96e-04 | gnorm=0.816 | tok/s=29,187 | MFU=7.0% | step=143702ms | tokens=0.492B | mem=4.2/32.7GB | cost=$242.79/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 07:12 (138.1h) | Phase3: SKIPPED | SFT: Oct 26 11:57 (142.8h) | DPO (Final): Oct 26 12:32 (143.4h) [2026-10-20 13:07:48][INFO][training.pretrain_engine] Starting global step 463 (accumulating 32 micro-batches)... [2026-10-20 13:07:48][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 13:08:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9550 [2026-10-20 13:08:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9911 [2026-10-20 13:08:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9977 [2026-10-20 13:08:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6640 [2026-10-20 13:08:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3310 [2026-10-20 13:08:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0044 [2026-10-20 13:08:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8889 [2026-10-20 13:08:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8665 [2026-10-20 13:08:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9942 [2026-10-20 13:08:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1627 [2026-10-20 13:08:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9300 [2026-10-20 13:08:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0997 [2026-10-20 13:08:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7124 [2026-10-20 13:08:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6118 [2026-10-20 13:08:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0082 [2026-10-20 13:08:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2429 [2026-10-20 13:09:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7390 [2026-10-20 13:09:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8612 [2026-10-20 13:09:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7028 [2026-10-20 13:09:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8854 [2026-10-20 13:09:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0144 [2026-10-20 13:09:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0046 [2026-10-20 13:09:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4035 [2026-10-20 13:09:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5953 [2026-10-20 13:09:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1409 [2026-10-20 13:09:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0179 [2026-10-20 13:09:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3220 [2026-10-20 13:09:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0436 [2026-10-20 13:10:06][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,561 | MFU: 1.8% [2026-10-20 13:10:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2729 [2026-10-20 13:10:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1483 [2026-10-20 13:10:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8245 [2026-10-20 13:10:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9579 [2026-10-20 13:10:11][INFO][training.pretrain_engine] [Ph2|Step 463] loss=6.0525 | ppl=425.18 | aux=0.02508 | lr=2.96e-04 | gnorm=1.219 | tok/s=29,188 | MFU=7.0% | step=143697ms | tokens=0.493B | mem=4.2/32.7GB | cost=$243.32/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 07:13 (138.1h) | Phase3: SKIPPED | SFT: Oct 26 11:59 (142.8h) | DPO (Final): Oct 26 12:33 (143.4h) [2026-10-20 13:10:11][INFO][training.pretrain_engine] Starting global step 464 (accumulating 32 micro-batches)... [2026-10-20 13:10:11][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 13:10:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6838 [2026-10-20 13:10:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9905 [2026-10-20 13:10:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1228 [2026-10-20 13:10:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8917 [2026-10-20 13:10:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4113 [2026-10-20 13:10:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9220 [2026-10-20 13:10:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0510 [2026-10-20 13:10:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0575 [2026-10-20 13:11:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7947 [2026-10-20 13:11:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9022 [2026-10-20 13:11:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9149 [2026-10-20 13:11:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8687 [2026-10-20 13:11:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9292 [2026-10-20 13:11:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9445 [2026-10-20 13:11:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0207 [2026-10-20 13:11:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1065 [2026-10-20 13:11:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5457 [2026-10-20 13:11:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7703 [2026-10-20 13:11:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8339 [2026-10-20 13:11:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8950 [2026-10-20 13:11:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1268 [2026-10-20 13:11:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1102 [2026-10-20 13:11:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1375 [2026-10-20 13:11:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0815 [2026-10-20 13:12:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6140 [2026-10-20 13:12:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0793 [2026-10-20 13:12:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6772 [2026-10-20 13:12:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8490 [2026-10-20 13:12:30][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,559 | MFU: 1.8% [2026-10-20 13:12:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1256 [2026-10-20 13:12:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1897 [2026-10-20 13:12:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8323 [2026-10-20 13:12:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1316 [2026-10-20 13:12:34][INFO][training.pretrain_engine] [Ph2|Step 464] loss=6.0513 | ppl=424.67 | aux=0.02509 | lr=2.97e-04 | gnorm=1.094 | tok/s=29,190 | MFU=7.0% | step=143691ms | tokens=0.494B | mem=4.2/32.7GB | cost=$243.84/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 07:15 (138.0h) | Phase3: SKIPPED | SFT: Oct 26 12:00 (142.8h) | DPO (Final): Oct 26 12:34 (143.4h) [2026-10-20 13:12:34][INFO][training.pretrain_engine] Starting global step 465 (accumulating 32 micro-batches)... [2026-10-20 13:12:34][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 13:12:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8040 [2026-10-20 13:12:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1715 [2026-10-20 13:12:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1289 [2026-10-20 13:12:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5374 [2026-10-20 13:13:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9430 [2026-10-20 13:13:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6260 [2026-10-20 13:13:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3289 [2026-10-20 13:13:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0037 [2026-10-20 13:13:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5974 [2026-10-20 13:13:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4468 [2026-10-20 13:13:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8493 [2026-10-20 13:13:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8408 [2026-10-20 13:13:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9465 [2026-10-20 13:13:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7550 [2026-10-20 13:13:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9359 [2026-10-20 13:13:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7501 [2026-10-20 13:14:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8350 [2026-10-20 13:14:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6504 [2026-10-20 13:14:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8230 [2026-10-20 13:14:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9504 [2026-10-20 13:14:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7715 [2026-10-20 13:14:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7474 [2026-10-20 13:14:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0936 [2026-10-20 13:14:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3765 [2026-10-20 13:14:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8893 [2026-10-20 13:14:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8688 [2026-10-20 13:14:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5813 [2026-10-20 13:14:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5445 [2026-10-20 13:14:53][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,557 | MFU: 1.8% [2026-10-20 13:14:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7603 [2026-10-20 13:14:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8626 [2026-10-20 13:14:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6091 [2026-10-20 13:14:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5430 [2026-10-20 13:14:57][INFO][training.pretrain_engine] [Ph2|Step 465] loss=6.0323 | ppl=416.68 | aux=0.02510 | lr=2.97e-04 | gnorm=1.141 | tok/s=29,191 | MFU=7.0% | step=143687ms | tokens=0.495B | mem=4.2/32.7GB | cost=$244.37/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 07:16 (138.0h) | Phase3: SKIPPED | SFT: Oct 26 12:02 (142.8h) | DPO (Final): Oct 26 12:36 (143.4h) [2026-10-20 13:14:57][INFO][training.pretrain_engine] Starting global step 466 (accumulating 32 micro-batches)... [2026-10-20 13:14:57][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 13:15:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4774 [2026-10-20 13:15:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3189 [2026-10-20 13:15:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2157 [2026-10-20 13:15:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6248 [2026-10-20 13:15:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9321 [2026-10-20 13:15:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0517 [2026-10-20 13:15:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0105 [2026-10-20 13:15:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3638 [2026-10-20 13:15:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0385 [2026-10-20 13:15:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4218 [2026-10-20 13:15:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8763 [2026-10-20 13:15:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7717 [2026-10-20 13:16:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3171 [2026-10-20 13:16:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6471 [2026-10-20 13:16:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9088 [2026-10-20 13:16:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0380 [2026-10-20 13:16:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7625 [2026-10-20 13:16:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4442 [2026-10-20 13:16:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8677 [2026-10-20 13:16:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7995 [2026-10-20 13:16:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4022 [2026-10-20 13:16:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8496 [2026-10-20 13:16:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9912 [2026-10-20 13:16:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0075 [2026-10-20 13:17:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2422 [2026-10-20 13:17:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3235 [2026-10-20 13:17:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1905 [2026-10-20 13:17:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9401 [2026-10-20 13:17:16][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,563 | MFU: 1.8% [2026-10-20 13:17:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8322 [2026-10-20 13:17:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2043 [2026-10-20 13:17:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9641 [2026-10-20 13:17:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7733 [2026-10-20 13:17:21][INFO][training.pretrain_engine] [Ph2|Step 466] loss=5.9078 | ppl=367.88 | aux=0.02512 | lr=2.98e-04 | gnorm=1.047 | tok/s=29,192 | MFU=7.0% | step=143680ms | tokens=0.496B | mem=4.2/32.7GB | cost=$244.89/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 07:18 (138.0h) | Phase3: SKIPPED | SFT: Oct 26 12:03 (142.8h) | DPO (Final): Oct 26 12:37 (143.3h) [2026-10-20 13:17:21][INFO][training.pretrain_engine] Starting global step 467 (accumulating 32 micro-batches)... [2026-10-20 13:17:21][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 13:17:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4777 [2026-10-20 13:17:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3243 [2026-10-20 13:17:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3107 [2026-10-20 13:17:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8793 [2026-10-20 13:17:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2008 [2026-10-20 13:17:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0389 [2026-10-20 13:17:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9015 [2026-10-20 13:17:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0565 [2026-10-20 13:18:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8898 [2026-10-20 13:18:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8348 [2026-10-20 13:18:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0056 [2026-10-20 13:18:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0734 [2026-10-20 13:18:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9609 [2026-10-20 13:18:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7563 [2026-10-20 13:18:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2936 [2026-10-20 13:18:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2143 [2026-10-20 13:18:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3690 [2026-10-20 13:18:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8408 [2026-10-20 13:18:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3481 [2026-10-20 13:18:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3715 [2026-10-20 13:19:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8503 [2026-10-20 13:19:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2620 [2026-10-20 13:19:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3366 [2026-10-20 13:19:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1001 [2026-10-20 13:19:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8210 [2026-10-20 13:19:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9697 [2026-10-20 13:19:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0253 [2026-10-20 13:19:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5222 [2026-10-20 13:19:38][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,624 | MFU: 1.8% [2026-10-20 13:19:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3237 [2026-10-20 13:19:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6580 [2026-10-20 13:19:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6753 [2026-10-20 13:19:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8554 [2026-10-20 13:19:44][INFO][training.pretrain_engine] [Ph2|Step 467] loss=5.9176 | ppl=371.52 | aux=0.02511 | lr=2.98e-04 | gnorm=1.055 | tok/s=29,194 | MFU=7.0% | step=143672ms | tokens=0.497B | mem=4.2/32.7GB | cost=$245.41/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 07:19 (138.0h) | Phase3: SKIPPED | SFT: Oct 26 12:04 (142.8h) | DPO (Final): Oct 26 12:39 (143.3h) [2026-10-20 13:19:44][INFO][training.pretrain_engine] Starting global step 468 (accumulating 32 micro-batches)... [2026-10-20 13:19:44][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 13:20:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9255 [2026-10-20 13:20:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5096 [2026-10-20 13:20:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9409 [2026-10-20 13:20:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9760 [2026-10-20 13:20:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7075 [2026-10-20 13:20:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8696 [2026-10-20 13:20:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3224 [2026-10-20 13:20:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7021 [2026-10-20 13:20:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0703 [2026-10-20 13:20:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8461 [2026-10-20 13:20:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7776 [2026-10-20 13:20:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4027 [2026-10-20 13:20:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.9063 [2026-10-20 13:20:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2964 [2026-10-20 13:20:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8441 [2026-10-20 13:20:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4821 [2026-10-20 13:21:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2751 [2026-10-20 13:21:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9686 [2026-10-20 13:21:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1903 [2026-10-20 13:21:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2564 [2026-10-20 13:21:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5123 [2026-10-20 13:21:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0954 [2026-10-20 13:21:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7518 [2026-10-20 13:21:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9662 [2026-10-20 13:21:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1784 [2026-10-20 13:21:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2109 [2026-10-20 13:21:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1065 [2026-10-20 13:21:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0097 [2026-10-20 13:22:02][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,603 | MFU: 1.8% [2026-10-20 13:22:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9606 [2026-10-20 13:22:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4855 [2026-10-20 13:22:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5350 [2026-10-20 13:22:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3880 [2026-10-20 13:22:07][INFO][training.pretrain_engine] [Ph2|Step 468] loss=6.0588 | ppl=427.87 | aux=0.02510 | lr=2.99e-04 | gnorm=1.383 | tok/s=29,196 | MFU=7.0% | step=143660ms | tokens=0.498B | mem=4.2/32.7GB | cost=$245.93/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 07:20 (138.0h) | Phase3: SKIPPED | SFT: Oct 26 12:06 (142.7h) | DPO (Final): Oct 26 12:40 (143.3h) [2026-10-20 13:22:10][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step468 | Tokens: 0.498B | Spent: $245.95 | Remaining: $154.05 [2026-10-20 13:22:10][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step468 [2026-10-20 13:22:10][INFO][training.pretrain_engine] Starting global step 469 (accumulating 32 micro-batches)... [2026-10-20 13:22:10][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 13:22:10][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 13:22:10][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 13:22:10][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 13:22:10][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step468 to Phase-Technologies/forge-500m-pretrain (path: phase2_step468)... [2026-10-20 13:22:10][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 13:22:19][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 13:22:19][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step468 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 13:22:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2256 [2026-10-20 13:22:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4892 [2026-10-20 13:22:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2285 [2026-10-20 13:22:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5223 [2026-10-20 13:22:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0838 [2026-10-20 13:22:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3817 [2026-10-20 13:22:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9856 [2026-10-20 13:22:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0543 [2026-10-20 13:23:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7096 [2026-10-20 13:23:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7970 [2026-10-20 13:23:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4942 [2026-10-20 13:23:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6958 [2026-10-20 13:23:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3054 [2026-10-20 13:23:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3528 [2026-10-20 13:23:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6657 [2026-10-20 13:23:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8646 [2026-10-20 13:23:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9947 [2026-10-20 13:23:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1161 [2026-10-20 13:23:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0232 [2026-10-20 13:23:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8805 [2026-10-20 13:23:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0149 [2026-10-20 13:23:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7670 [2026-10-20 13:23:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1330 [2026-10-20 13:23:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6958 [2026-10-20 13:24:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8405 [2026-10-20 13:24:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6886 [2026-10-20 13:24:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0475 [2026-10-20 13:24:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5376 [2026-10-20 13:24:28][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,574 | MFU: 1.8% [2026-10-20 13:24:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3671 [2026-10-20 13:24:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2822 [2026-10-20 13:24:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8313 [2026-10-20 13:24:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0524 [2026-10-20 13:24:33][INFO][training.pretrain_engine] [Ph2|Step 469] loss=5.8551 | ppl=349.00 | aux=0.02511 | lr=3.00e-04 | gnorm=0.789 | tok/s=29,198 | MFU=7.0% | step=143652ms | tokens=0.499B | mem=4.2/32.7GB | cost=$246.47/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 07:21 (138.0h) | Phase3: SKIPPED | SFT: Oct 26 12:07 (142.7h) | DPO (Final): Oct 26 12:41 (143.3h) [2026-10-20 13:24:33][INFO][training.pretrain_engine] Starting global step 470 (accumulating 32 micro-batches)... [2026-10-20 13:24:33][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 13:24:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6881 [2026-10-20 13:24:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5595 [2026-10-20 13:24:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8375 [2026-10-20 13:24:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9716 [2026-10-20 13:25:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9270 [2026-10-20 13:25:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3419 [2026-10-20 13:25:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4027 [2026-10-20 13:25:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8997 [2026-10-20 13:25:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0619 [2026-10-20 13:25:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3489 [2026-10-20 13:25:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5443 [2026-10-20 13:25:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3169 [2026-10-20 13:25:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2520 [2026-10-20 13:25:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3953 [2026-10-20 13:25:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9850 [2026-10-20 13:25:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8989 [2026-10-20 13:26:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1845 [2026-10-20 13:26:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7526 [2026-10-20 13:26:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1343 [2026-10-20 13:26:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7921 [2026-10-20 13:26:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6184 [2026-10-20 13:26:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0659 [2026-10-20 13:26:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7334 [2026-10-20 13:26:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4189 [2026-10-20 13:26:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5125 [2026-10-20 13:26:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5132 [2026-10-20 13:26:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3187 [2026-10-20 13:26:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3903 [2026-10-20 13:26:51][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,563 | MFU: 1.8% [2026-10-20 13:26:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7117 [2026-10-20 13:26:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5245 [2026-10-20 13:26:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2128 [2026-10-20 13:26:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4797 [2026-10-20 13:26:56][INFO][training.pretrain_engine] [Ph2|Step 470] loss=6.0975 | ppl=444.75 | aux=0.02511 | lr=3.00e-04 | gnorm=1.094 | tok/s=29,198 | MFU=7.0% | step=143651ms | tokens=0.500B | mem=4.2/32.7GB | cost=$246.99/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 07:23 (137.9h) | Phase3: SKIPPED | SFT: Oct 26 12:09 (142.7h) | DPO (Final): Oct 26 12:43 (143.3h) [2026-10-20 13:26:59][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step470 | Tokens: 0.500B | Spent: $247.00 | Remaining: $153.00 [2026-10-20 13:26:59][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step470 [2026-10-20 13:26:59][INFO][training.pretrain_engine] Starting global step 471 (accumulating 32 micro-batches)... [2026-10-20 13:26:59][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 13:26:59][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 13:26:59][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 13:26:59][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 13:26:59][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step470 to Phase-Technologies/forge-500m-pretrain (path: phase2_step470)... [2026-10-20 13:26:59][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 13:27:09][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 13:27:09][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step470 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 13:27:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7909 [2026-10-20 13:27:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8016 [2026-10-20 13:27:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9611 [2026-10-20 13:27:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1224 [2026-10-20 13:27:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6872 [2026-10-20 13:27:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4293 [2026-10-20 13:27:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9272 [2026-10-20 13:27:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5025 [2026-10-20 13:27:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5215 [2026-10-20 13:27:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0033 [2026-10-20 13:27:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0430 [2026-10-20 13:27:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0188 [2026-10-20 13:28:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1203 [2026-10-20 13:28:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1346 [2026-10-20 13:28:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7692 [2026-10-20 13:28:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0623 [2026-10-20 13:28:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3624 [2026-10-20 13:28:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1690 [2026-10-20 13:28:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0201 [2026-10-20 13:28:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9707 [2026-10-20 13:28:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1593 [2026-10-20 13:28:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7808 [2026-10-20 13:28:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2942 [2026-10-20 13:28:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9644 [2026-10-20 13:29:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7817 [2026-10-20 13:29:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7789 [2026-10-20 13:29:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4003 [2026-10-20 13:29:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6265 [2026-10-20 13:29:18][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,564 | MFU: 1.8% [2026-10-20 13:29:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0664 [2026-10-20 13:29:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9322 [2026-10-20 13:29:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1411 [2026-10-20 13:29:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4222 [2026-10-20 13:29:22][INFO][training.pretrain_engine] [Ph2|Step 471] loss=6.0299 | ppl=415.66 | aux=0.02510 | lr=3.00e-04 | gnorm=1.523 | tok/s=29,198 | MFU=7.0% | step=143649ms | tokens=0.501B | mem=4.2/32.7GB | cost=$247.53/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 07:25 (137.9h) | Phase3: SKIPPED | SFT: Oct 26 12:10 (142.7h) | DPO (Final): Oct 26 12:45 (143.3h) [2026-10-20 13:29:22][INFO][training.pretrain_engine] Starting global step 472 (accumulating 32 micro-batches)... [2026-10-20 13:29:22][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 13:29:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2105 [2026-10-20 13:29:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8530 [2026-10-20 13:29:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0013 [2026-10-20 13:29:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7183 [2026-10-20 13:29:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8123 [2026-10-20 13:29:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5700 [2026-10-20 13:29:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5868 [2026-10-20 13:29:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2695 [2026-10-20 13:30:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0259 [2026-10-20 13:30:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7376 [2026-10-20 13:30:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3065 [2026-10-20 13:30:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6620 [2026-10-20 13:30:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4203 [2026-10-20 13:30:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9627 [2026-10-20 13:30:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7052 [2026-10-20 13:30:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7071 [2026-10-20 13:30:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6755 [2026-10-20 13:30:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0244 [2026-10-20 13:30:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4154 [2026-10-20 13:30:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9939 [2026-10-20 13:31:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1164 [2026-10-20 13:31:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1288 [2026-10-20 13:31:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0365 [2026-10-20 13:31:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1379 [2026-10-20 13:31:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4325 [2026-10-20 13:31:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3545 [2026-10-20 13:31:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9890 [2026-10-20 13:31:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2207 [2026-10-20 13:31:41][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,548 | MFU: 1.8% [2026-10-20 13:31:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6455 [2026-10-20 13:31:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8836 [2026-10-20 13:31:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2248 [2026-10-20 13:31:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0673 [2026-10-20 13:31:46][INFO][training.pretrain_engine] [Ph2|Step 472] loss=5.9650 | ppl=389.57 | aux=0.02509 | lr=3.00e-04 | gnorm=0.973 | tok/s=29,200 | MFU=7.0% | step=143641ms | tokens=0.502B | mem=4.2/32.7GB | cost=$248.05/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 07:26 (137.9h) | Phase3: SKIPPED | SFT: Oct 26 12:12 (142.7h) | DPO (Final): Oct 26 12:46 (143.2h) [2026-10-20 13:31:46][INFO][training.pretrain_engine] Starting global step 473 (accumulating 32 micro-batches)... [2026-10-20 13:31:46][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 13:32:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4059 [2026-10-20 13:32:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6561 [2026-10-20 13:32:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5933 [2026-10-20 13:32:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2015 [2026-10-20 13:32:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1672 [2026-10-20 13:32:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4125 [2026-10-20 13:32:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2030 [2026-10-20 13:32:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3018 [2026-10-20 13:32:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6453 [2026-10-20 13:32:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2158 [2026-10-20 13:32:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1701 [2026-10-20 13:32:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3485 [2026-10-20 13:32:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5970 [2026-10-20 13:32:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3965 [2026-10-20 13:32:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8768 [2026-10-20 13:32:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2590 [2026-10-20 13:33:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8144 [2026-10-20 13:33:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0750 [2026-10-20 13:33:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7773 [2026-10-20 13:33:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3252 [2026-10-20 13:33:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5598 [2026-10-20 13:33:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6763 [2026-10-20 13:33:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7862 [2026-10-20 13:33:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9479 [2026-10-20 13:33:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2130 [2026-10-20 13:33:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5157 [2026-10-20 13:33:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6248 [2026-10-20 13:33:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5687 [2026-10-20 13:34:04][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,560 | MFU: 1.8% [2026-10-20 13:34:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7984 [2026-10-20 13:34:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0569 [2026-10-20 13:34:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9184 [2026-10-20 13:34:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1023 [2026-10-20 13:34:09][INFO][training.pretrain_engine] [Ph2|Step 473] loss=5.9476 | ppl=382.82 | aux=0.02511 | lr=3.00e-04 | gnorm=1.461 | tok/s=29,202 | MFU=7.0% | step=143633ms | tokens=0.503B | mem=4.2/32.7GB | cost=$248.57/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 07:28 (137.9h) | Phase3: SKIPPED | SFT: Oct 26 12:13 (142.7h) | DPO (Final): Oct 26 12:47 (143.2h) [2026-10-20 13:34:09][INFO][training.pretrain_engine] Starting global step 474 (accumulating 32 micro-batches)... [2026-10-20 13:34:09][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 13:34:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3995 [2026-10-20 13:34:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1826 [2026-10-20 13:34:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3234 [2026-10-20 13:34:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7990 [2026-10-20 13:34:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9687 [2026-10-20 13:34:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7449 [2026-10-20 13:34:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8187 [2026-10-20 13:34:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0179 [2026-10-20 13:35:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1511 [2026-10-20 13:35:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0624 [2026-10-20 13:35:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0849 [2026-10-20 13:35:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8004 [2026-10-20 13:35:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9214 [2026-10-20 13:35:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5985 [2026-10-20 13:35:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0119 [2026-10-20 13:35:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0206 [2026-10-20 13:35:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6388 [2026-10-20 13:35:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8552 [2026-10-20 13:35:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8397 [2026-10-20 13:35:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6802 [2026-10-20 13:35:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7972 [2026-10-20 13:35:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9830 [2026-10-20 13:35:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7166 [2026-10-20 13:35:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7135 [2026-10-20 13:36:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1425 [2026-10-20 13:36:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2760 [2026-10-20 13:36:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6630 [2026-10-20 13:36:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1575 [2026-10-20 13:36:28][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,555 | MFU: 1.8% [2026-10-20 13:36:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1348 [2026-10-20 13:36:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2106 [2026-10-20 13:36:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4646 [2026-10-20 13:36:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2328 [2026-10-20 13:36:32][INFO][training.pretrain_engine] [Ph2|Step 474] loss=6.0058 | ppl=405.77 | aux=0.02509 | lr=3.00e-04 | gnorm=0.816 | tok/s=29,203 | MFU=7.0% | step=143626ms | tokens=0.504B | mem=4.2/32.7GB | cost=$249.10/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 07:29 (137.9h) | Phase3: SKIPPED | SFT: Oct 26 12:14 (142.6h) | DPO (Final): Oct 26 12:49 (143.2h) [2026-10-20 13:36:32][INFO][training.pretrain_engine] Starting global step 475 (accumulating 32 micro-batches)... [2026-10-20 13:36:32][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 13:36:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7480 [2026-10-20 13:36:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7258 [2026-10-20 13:36:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5062 [2026-10-20 13:36:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4995 [2026-10-20 13:37:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4070 [2026-10-20 13:37:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7659 [2026-10-20 13:37:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2931 [2026-10-20 13:37:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5429 [2026-10-20 13:37:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9253 [2026-10-20 13:37:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8934 [2026-10-20 13:37:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1734 [2026-10-20 13:37:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1966 [2026-10-20 13:37:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6427 [2026-10-20 13:37:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1312 [2026-10-20 13:37:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3486 [2026-10-20 13:37:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8530 [2026-10-20 13:38:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2546 [2026-10-20 13:38:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2135 [2026-10-20 13:38:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9236 [2026-10-20 13:38:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0583 [2026-10-20 13:38:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2059 [2026-10-20 13:38:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8516 [2026-10-20 13:38:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0711 [2026-10-20 13:38:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8706 [2026-10-20 13:38:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9400 [2026-10-20 13:38:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1969 [2026-10-20 13:38:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5582 [2026-10-20 13:38:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1949 [2026-10-20 13:38:51][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,558 | MFU: 1.8% [2026-10-20 13:38:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9883 [2026-10-20 13:38:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3804 [2026-10-20 13:38:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9899 [2026-10-20 13:38:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7781 [2026-10-20 13:38:55][INFO][training.pretrain_engine] [Ph2|Step 475] loss=6.0881 | ppl=440.60 | aux=0.02510 | lr=3.00e-04 | gnorm=1.578 | tok/s=29,205 | MFU=7.0% | step=143618ms | tokens=0.505B | mem=4.2/32.7GB | cost=$249.62/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 07:30 (137.9h) | Phase3: SKIPPED | SFT: Oct 26 12:16 (142.6h) | DPO (Final): Oct 26 12:50 (143.2h) [2026-10-20 13:38:55][INFO][training.pretrain_engine] Starting global step 476 (accumulating 32 micro-batches)... [2026-10-20 13:38:55][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 13:39:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7087 [2026-10-20 13:39:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4053 [2026-10-20 13:39:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0117 [2026-10-20 13:39:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2539 [2026-10-20 13:39:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1306 [2026-10-20 13:39:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1890 [2026-10-20 13:39:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9024 [2026-10-20 13:39:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2504 [2026-10-20 13:39:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3567 [2026-10-20 13:39:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1870 [2026-10-20 13:39:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1039 [2026-10-20 13:39:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5476 [2026-10-20 13:40:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8295 [2026-10-20 13:40:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2430 [2026-10-20 13:40:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5579 [2026-10-20 13:40:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2678 [2026-10-20 13:40:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2464 [2026-10-20 13:40:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4479 [2026-10-20 13:40:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7070 [2026-10-20 13:40:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4944 [2026-10-20 13:40:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1866 [2026-10-20 13:40:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1350 [2026-10-20 13:40:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5044 [2026-10-20 13:40:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1978 [2026-10-20 13:41:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9950 [2026-10-20 13:41:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8298 [2026-10-20 13:41:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1951 [2026-10-20 13:41:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0492 [2026-10-20 13:41:14][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,552 | MFU: 1.8% [2026-10-20 13:41:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3100 [2026-10-20 13:41:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7703 [2026-10-20 13:41:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9010 [2026-10-20 13:41:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0286 [2026-10-20 13:41:19][INFO][training.pretrain_engine] [Ph2|Step 476] loss=5.9856 | ppl=397.65 | aux=0.02510 | lr=3.00e-04 | gnorm=1.125 | tok/s=29,205 | MFU=7.0% | step=143615ms | tokens=0.506B | mem=4.2/32.7GB | cost=$250.15/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 07:32 (137.9h) | Phase3: SKIPPED | SFT: Oct 26 12:17 (142.6h) | DPO (Final): Oct 26 12:51 (143.2h) [2026-10-20 13:41:19][INFO][training.pretrain_engine] Starting global step 477 (accumulating 32 micro-batches)... [2026-10-20 13:41:19][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 13:41:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8913 [2026-10-20 13:41:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1892 [2026-10-20 13:41:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0812 [2026-10-20 13:41:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2870 [2026-10-20 13:41:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3574 [2026-10-20 13:41:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9045 [2026-10-20 13:41:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8120 [2026-10-20 13:41:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0604 [2026-10-20 13:42:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1491 [2026-10-20 13:42:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2344 [2026-10-20 13:42:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8567 [2026-10-20 13:42:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0212 [2026-10-20 13:42:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1727 [2026-10-20 13:42:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0579 [2026-10-20 13:42:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5493 [2026-10-20 13:42:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1623 [2026-10-20 13:42:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0559 [2026-10-20 13:42:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2804 [2026-10-20 13:42:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0674 [2026-10-20 13:42:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8279 [2026-10-20 13:43:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8695 [2026-10-20 13:43:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0207 [2026-10-20 13:43:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9880 [2026-10-20 13:43:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9623 [2026-10-20 13:43:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6494 [2026-10-20 13:43:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7259 [2026-10-20 13:43:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2241 [2026-10-20 13:43:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6770 [2026-10-20 13:43:37][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,582 | MFU: 1.8% [2026-10-20 13:43:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9786 [2026-10-20 13:43:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9994 [2026-10-20 13:43:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5889 [2026-10-20 13:43:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9249 [2026-10-20 13:43:42][INFO][training.pretrain_engine] [Ph2|Step 477] loss=5.9849 | ppl=397.39 | aux=0.02510 | lr=3.00e-04 | gnorm=1.375 | tok/s=29,207 | MFU=7.0% | step=143608ms | tokens=0.508B | mem=4.2/32.7GB | cost=$250.67/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 07:33 (137.8h) | Phase3: SKIPPED | SFT: Oct 26 12:19 (142.6h) | DPO (Final): Oct 26 12:53 (143.2h) [2026-10-20 13:43:42][INFO][training.pretrain_engine] Starting global step 478 (accumulating 32 micro-batches)... [2026-10-20 13:43:42][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 13:44:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9070 [2026-10-20 13:44:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9769 [2026-10-20 13:44:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1315 [2026-10-20 13:44:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8668 [2026-10-20 13:44:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5566 [2026-10-20 13:44:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9503 [2026-10-20 13:44:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4524 [2026-10-20 13:44:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7874 [2026-10-20 13:44:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5204 [2026-10-20 13:44:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1425 [2026-10-20 13:44:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6778 [2026-10-20 13:44:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1380 [2026-10-20 13:44:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7776 [2026-10-20 13:44:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8186 [2026-10-20 13:44:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3481 [2026-10-20 13:44:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6310 [2026-10-20 13:45:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5099 [2026-10-20 13:45:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9110 [2026-10-20 13:45:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3999 [2026-10-20 13:45:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4864 [2026-10-20 13:45:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8859 [2026-10-20 13:45:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9077 [2026-10-20 13:45:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.1458 [2026-10-20 13:45:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4849 [2026-10-20 13:45:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1491 [2026-10-20 13:45:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3694 [2026-10-20 13:45:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9140 [2026-10-20 13:45:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4247 [2026-10-20 13:46:00][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,569 | MFU: 1.8% [2026-10-20 13:46:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1691 [2026-10-20 13:46:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8039 [2026-10-20 13:46:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7790 [2026-10-20 13:46:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4959 [2026-10-20 13:46:05][INFO][training.pretrain_engine] [Ph2|Step 478] loss=6.0009 | ppl=403.77 | aux=0.02510 | lr=3.00e-04 | gnorm=1.039 | tok/s=29,208 | MFU=7.0% | step=143600ms | tokens=0.509B | mem=4.2/32.7GB | cost=$251.19/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 07:35 (137.8h) | Phase3: SKIPPED | SFT: Oct 26 12:20 (142.6h) | DPO (Final): Oct 26 12:54 (143.1h) [2026-10-20 13:46:05][INFO][training.pretrain_engine] Starting global step 479 (accumulating 32 micro-batches)... [2026-10-20 13:46:05][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 13:46:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1788 [2026-10-20 13:46:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1097 [2026-10-20 13:46:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1680 [2026-10-20 13:46:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9897 [2026-10-20 13:46:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8531 [2026-10-20 13:46:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7438 [2026-10-20 13:46:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6774 [2026-10-20 13:46:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0381 [2026-10-20 13:46:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1427 [2026-10-20 13:46:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1819 [2026-10-20 13:46:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4176 [2026-10-20 13:46:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0364 [2026-10-20 13:47:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9142 [2026-10-20 13:47:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4744 [2026-10-20 13:47:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.8281 [2026-10-20 13:47:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5898 [2026-10-20 13:47:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8486 [2026-10-20 13:47:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8405 [2026-10-20 13:47:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8407 [2026-10-20 13:47:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3140 [2026-10-20 13:47:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9959 [2026-10-20 13:47:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5306 [2026-10-20 13:47:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9846 [2026-10-20 13:47:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7056 [2026-10-20 13:48:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6679 [2026-10-20 13:48:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1102 [2026-10-20 13:48:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9531 [2026-10-20 13:48:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1385 [2026-10-20 13:48:24][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,554 | MFU: 1.8% [2026-10-20 13:48:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0792 [2026-10-20 13:48:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8344 [2026-10-20 13:48:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1147 [2026-10-20 13:48:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7717 [2026-10-20 13:48:28][INFO][training.pretrain_engine] [Ph2|Step 479] loss=6.0113 | ppl=408.00 | aux=0.02511 | lr=3.00e-04 | gnorm=1.109 | tok/s=29,210 | MFU=7.0% | step=143589ms | tokens=0.510B | mem=4.2/32.7GB | cost=$251.72/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 07:36 (137.8h) | Phase3: SKIPPED | SFT: Oct 26 12:21 (142.6h) | DPO (Final): Oct 26 12:55 (143.1h) [2026-10-20 13:48:31][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step479 | Tokens: 0.510B | Spent: $251.73 | Remaining: $148.27 [2026-10-20 13:48:31][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step479 [2026-10-20 13:48:31][INFO][training.pretrain_engine] Starting global step 480 (accumulating 32 micro-batches)... [2026-10-20 13:48:31][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 13:48:32][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 13:48:32][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 13:48:32][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 13:48:32][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step479 to Phase-Technologies/forge-500m-pretrain (path: phase2_step479)... [2026-10-20 13:48:32][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 13:48:41][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 13:48:41][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step479 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 13:48:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5131 [2026-10-20 13:48:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1463 [2026-10-20 13:48:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2384 [2026-10-20 13:48:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3988 [2026-10-20 13:49:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6776 [2026-10-20 13:49:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9208 [2026-10-20 13:49:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1506 [2026-10-20 13:49:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9800 [2026-10-20 13:49:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9431 [2026-10-20 13:49:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8409 [2026-10-20 13:49:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9127 [2026-10-20 13:49:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5941 [2026-10-20 13:49:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3645 [2026-10-20 13:49:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6027 [2026-10-20 13:49:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0912 [2026-10-20 13:49:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8359 [2026-10-20 13:49:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9970 [2026-10-20 13:49:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9140 [2026-10-20 13:49:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1528 [2026-10-20 13:50:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0428 [2026-10-20 13:50:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8618 [2026-10-20 13:50:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2118 [2026-10-20 13:50:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7573 [2026-10-20 13:50:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7359 [2026-10-20 13:50:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1389 [2026-10-20 13:50:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1333 [2026-10-20 13:50:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6757 [2026-10-20 13:50:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3833 [2026-10-20 13:50:50][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,561 | MFU: 1.8% [2026-10-20 13:50:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2611 [2026-10-20 13:50:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8246 [2026-10-20 13:50:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3817 [2026-10-20 13:50:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7774 [2026-10-20 13:50:55][INFO][training.pretrain_engine] [Ph2|Step 480] loss=6.0424 | ppl=420.91 | aux=0.02512 | lr=3.00e-04 | gnorm=1.023 | tok/s=29,212 | MFU=7.0% | step=143579ms | tokens=0.511B | mem=4.2/32.7GB | cost=$252.25/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 07:37 (137.8h) | Phase3: SKIPPED | SFT: Oct 26 12:22 (142.5h) | DPO (Final): Oct 26 12:57 (143.1h) [2026-10-20 13:50:55][INFO][training.pretrain_engine] Starting global step 481 (accumulating 32 micro-batches)... [2026-10-20 13:50:55][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 13:51:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=7.6746 [2026-10-20 13:51:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9423 [2026-10-20 13:51:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9612 [2026-10-20 13:51:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8896 [2026-10-20 13:51:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1059 [2026-10-20 13:51:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6595 [2026-10-20 13:51:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7667 [2026-10-20 13:51:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8852 [2026-10-20 13:51:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0263 [2026-10-20 13:51:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9490 [2026-10-20 13:51:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1223 [2026-10-20 13:51:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7415 [2026-10-20 13:52:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3548 [2026-10-20 13:52:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1825 [2026-10-20 13:52:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4115 [2026-10-20 13:52:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2039 [2026-10-20 13:52:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6570 [2026-10-20 13:52:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9641 [2026-10-20 13:52:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3195 [2026-10-20 13:52:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4116 [2026-10-20 13:52:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1667 [2026-10-20 13:52:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0032 [2026-10-20 13:52:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6925 [2026-10-20 13:52:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9248 [2026-10-20 13:52:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8203 [2026-10-20 13:53:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9961 [2026-10-20 13:53:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7958 [2026-10-20 13:53:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0941 [2026-10-20 13:53:13][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,580 | MFU: 1.8% [2026-10-20 13:53:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8782 [2026-10-20 13:53:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6310 [2026-10-20 13:53:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4408 [2026-10-20 13:53:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5549 [2026-10-20 13:53:18][INFO][training.pretrain_engine] [Ph2|Step 481] loss=5.8897 | ppl=361.29 | aux=0.02510 | lr=3.00e-04 | gnorm=0.980 | tok/s=29,213 | MFU=7.0% | step=143574ms | tokens=0.512B | mem=4.2/32.7GB | cost=$252.78/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 07:39 (137.8h) | Phase3: SKIPPED | SFT: Oct 26 12:24 (142.5h) | DPO (Final): Oct 26 12:58 (143.1h) [2026-10-20 13:53:18][INFO][training.pretrain_engine] Starting global step 482 (accumulating 32 micro-batches)... [2026-10-20 13:53:18][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 13:53:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3791 [2026-10-20 13:53:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0014 [2026-10-20 13:53:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2476 [2026-10-20 13:53:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1018 [2026-10-20 13:53:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0330 [2026-10-20 13:53:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1436 [2026-10-20 13:53:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6979 [2026-10-20 13:53:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7791 [2026-10-20 13:54:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9975 [2026-10-20 13:54:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8672 [2026-10-20 13:54:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3342 [2026-10-20 13:54:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3762 [2026-10-20 13:54:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0099 [2026-10-20 13:54:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1747 [2026-10-20 13:54:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3200 [2026-10-20 13:54:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8636 [2026-10-20 13:54:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6626 [2026-10-20 13:54:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6734 [2026-10-20 13:54:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1419 [2026-10-20 13:54:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7599 [2026-10-20 13:55:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.2468 [2026-10-20 13:55:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3302 [2026-10-20 13:55:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3153 [2026-10-20 13:55:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7353 [2026-10-20 13:55:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6219 [2026-10-20 13:55:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8104 [2026-10-20 13:55:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0324 [2026-10-20 13:55:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5200 [2026-10-20 13:55:37][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,528 | MFU: 1.8% [2026-10-20 13:55:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0493 [2026-10-20 13:55:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8671 [2026-10-20 13:55:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0866 [2026-10-20 13:55:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8106 [2026-10-20 13:55:42][INFO][training.pretrain_engine] [Ph2|Step 482] loss=5.9118 | ppl=369.39 | aux=0.02511 | lr=3.00e-04 | gnorm=1.078 | tok/s=29,214 | MFU=7.0% | step=143573ms | tokens=0.513B | mem=4.2/32.7GB | cost=$253.30/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 07:40 (137.8h) | Phase3: SKIPPED | SFT: Oct 26 12:26 (142.5h) | DPO (Final): Oct 26 13:00 (143.1h) [2026-10-20 13:55:42][INFO][training.pretrain_engine] Starting global step 483 (accumulating 32 micro-batches)... [2026-10-20 13:55:42][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 13:55:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1940 [2026-10-20 13:55:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9251 [2026-10-20 13:55:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8970 [2026-10-20 13:56:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0703 [2026-10-20 13:56:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0157 [2026-10-20 13:56:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8413 [2026-10-20 13:56:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0921 [2026-10-20 13:56:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8984 [2026-10-20 13:56:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5802 [2026-10-20 13:56:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0239 [2026-10-20 13:56:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3046 [2026-10-20 13:56:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1819 [2026-10-20 13:56:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7370 [2026-10-20 13:56:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8044 [2026-10-20 13:56:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7937 [2026-10-20 13:56:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6676 [2026-10-20 13:57:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7303 [2026-10-20 13:57:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2073 [2026-10-20 13:57:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1050 [2026-10-20 13:57:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2332 [2026-10-20 13:57:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6212 [2026-10-20 13:57:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3864 [2026-10-20 13:57:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6092 [2026-10-20 13:57:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3074 [2026-10-20 13:57:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0179 [2026-10-20 13:57:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0618 [2026-10-20 13:57:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7877 [2026-10-20 13:57:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8154 [2026-10-20 13:58:00][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,569 | MFU: 1.8% [2026-10-20 13:58:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1474 [2026-10-20 13:58:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4256 [2026-10-20 13:58:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8169 [2026-10-20 13:58:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0963 [2026-10-20 13:58:05][INFO][training.pretrain_engine] [Ph2|Step 483] loss=6.0618 | ppl=429.14 | aux=0.02513 | lr=3.00e-04 | gnorm=1.469 | tok/s=29,214 | MFU=7.0% | step=143570ms | tokens=0.514B | mem=4.2/32.7GB | cost=$253.83/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 07:42 (137.7h) | Phase3: SKIPPED | SFT: Oct 26 12:27 (142.5h) | DPO (Final): Oct 26 13:01 (143.1h) [2026-10-20 13:58:05][INFO][training.pretrain_engine] Starting global step 484 (accumulating 32 micro-batches)... [2026-10-20 13:58:05][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 13:58:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5783 [2026-10-20 13:58:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9661 [2026-10-20 13:58:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2645 [2026-10-20 13:58:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9150 [2026-10-20 13:58:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6833 [2026-10-20 13:58:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3468 [2026-10-20 13:58:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7437 [2026-10-20 13:58:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2863 [2026-10-20 13:58:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2237 [2026-10-20 13:58:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3240 [2026-10-20 13:58:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2409 [2026-10-20 13:58:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2593 [2026-10-20 13:59:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5315 [2026-10-20 13:59:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0733 [2026-10-20 13:59:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7790 [2026-10-20 13:59:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6733 [2026-10-20 13:59:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6982 [2026-10-20 13:59:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1837 [2026-10-20 13:59:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3790 [2026-10-20 13:59:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0428 [2026-10-20 13:59:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0519 [2026-10-20 13:59:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2988 [2026-10-20 13:59:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2318 [2026-10-20 13:59:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2662 [2026-10-20 14:00:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4845 [2026-10-20 14:00:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9551 [2026-10-20 14:00:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1526 [2026-10-20 14:00:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1113 [2026-10-20 14:00:24][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,568 | MFU: 1.8% [2026-10-20 14:00:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1810 [2026-10-20 14:00:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8394 [2026-10-20 14:00:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9736 [2026-10-20 14:00:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4802 [2026-10-20 14:00:28][INFO][training.pretrain_engine] [Ph2|Step 484] loss=5.9288 | ppl=375.69 | aux=0.02512 | lr=3.00e-04 | gnorm=0.895 | tok/s=29,216 | MFU=7.0% | step=143561ms | tokens=0.515B | mem=4.2/32.7GB | cost=$254.35/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 07:43 (137.7h) | Phase3: SKIPPED | SFT: Oct 26 12:28 (142.5h) | DPO (Final): Oct 26 13:03 (143.0h) [2026-10-20 14:00:28][INFO][training.pretrain_engine] Starting global step 485 (accumulating 32 micro-batches)... [2026-10-20 14:00:28][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 14:00:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0149 [2026-10-20 14:00:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1617 [2026-10-20 14:00:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0486 [2026-10-20 14:00:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8569 [2026-10-20 14:01:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8124 [2026-10-20 14:01:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4575 [2026-10-20 14:01:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6376 [2026-10-20 14:01:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5291 [2026-10-20 14:01:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0252 [2026-10-20 14:01:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7802 [2026-10-20 14:01:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7598 [2026-10-20 14:01:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0611 [2026-10-20 14:01:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1350 [2026-10-20 14:01:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3751 [2026-10-20 14:01:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2202 [2026-10-20 14:01:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1111 [2026-10-20 14:01:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0983 [2026-10-20 14:01:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8676 [2026-10-20 14:01:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5704 [2026-10-20 14:01:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4685 [2026-10-20 14:02:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5308 [2026-10-20 14:02:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6344 [2026-10-20 14:02:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7970 [2026-10-20 14:02:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8408 [2026-10-20 14:02:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0255 [2026-10-20 14:02:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4482 [2026-10-20 14:02:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2414 [2026-10-20 14:02:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1646 [2026-10-20 14:02:47][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,554 | MFU: 1.8% [2026-10-20 14:02:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8743 [2026-10-20 14:02:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4282 [2026-10-20 14:02:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7362 [2026-10-20 14:02:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0440 [2026-10-20 14:02:51][INFO][training.pretrain_engine] [Ph2|Step 485] loss=6.0103 | ppl=407.59 | aux=0.02509 | lr=3.00e-04 | gnorm=1.422 | tok/s=29,217 | MFU=7.0% | step=143555ms | tokens=0.516B | mem=4.2/32.7GB | cost=$254.87/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 07:45 (137.7h) | Phase3: SKIPPED | SFT: Oct 26 12:30 (142.5h) | DPO (Final): Oct 26 13:04 (143.0h) [2026-10-20 14:02:51][INFO][training.pretrain_engine] Starting global step 486 (accumulating 32 micro-batches)... [2026-10-20 14:02:51][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 14:03:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2380 [2026-10-20 14:03:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7764 [2026-10-20 14:03:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9692 [2026-10-20 14:03:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9353 [2026-10-20 14:03:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3988 [2026-10-20 14:03:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6588 [2026-10-20 14:03:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3380 [2026-10-20 14:03:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5771 [2026-10-20 14:03:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7007 [2026-10-20 14:03:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4220 [2026-10-20 14:03:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7776 [2026-10-20 14:03:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7635 [2026-10-20 14:04:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8028 [2026-10-20 14:04:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5552 [2026-10-20 14:04:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6975 [2026-10-20 14:04:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8660 [2026-10-20 14:04:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1508 [2026-10-20 14:04:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4243 [2026-10-20 14:04:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6981 [2026-10-20 14:04:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5645 [2026-10-20 14:04:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0823 [2026-10-20 14:04:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0503 [2026-10-20 14:04:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3701 [2026-10-20 14:04:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0199 [2026-10-20 14:04:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1971 [2026-10-20 14:04:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5019 [2026-10-20 14:04:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3082 [2026-10-20 14:04:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3330 [2026-10-20 14:05:12][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,452 | MFU: 1.8% [2026-10-20 14:05:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.7969 [2026-10-20 14:05:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2784 [2026-10-20 14:05:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6494 [2026-10-20 14:05:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6283 [2026-10-20 14:05:17][INFO][training.pretrain_engine] [Ph2|Step 486] loss=5.9626 | ppl=388.63 | aux=0.02511 | lr=3.00e-04 | gnorm=0.828 | tok/s=29,215 | MFU=7.0% | step=143567ms | tokens=0.517B | mem=4.2/32.7GB | cost=$255.40/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 07:47 (137.7h) | Phase3: SKIPPED | SFT: Oct 26 12:32 (142.5h) | DPO (Final): Oct 26 13:07 (143.0h) [2026-10-20 14:05:17][INFO][training.pretrain_engine] Starting global step 487 (accumulating 32 micro-batches)... [2026-10-20 14:05:17][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 14:05:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2719 [2026-10-20 14:05:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3512 [2026-10-20 14:05:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0486 [2026-10-20 14:05:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4208 [2026-10-20 14:05:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8764 [2026-10-20 14:05:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2521 [2026-10-20 14:05:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1188 [2026-10-20 14:05:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3963 [2026-10-20 14:06:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9199 [2026-10-20 14:06:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1116 [2026-10-20 14:06:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0005 [2026-10-20 14:06:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8126 [2026-10-20 14:06:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0658 [2026-10-20 14:06:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9197 [2026-10-20 14:06:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7231 [2026-10-20 14:06:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5493 [2026-10-20 14:06:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6361 [2026-10-20 14:06:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1060 [2026-10-20 14:06:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7336 [2026-10-20 14:06:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2243 [2026-10-20 14:07:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9877 [2026-10-20 14:07:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8212 [2026-10-20 14:07:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7833 [2026-10-20 14:07:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9068 [2026-10-20 14:07:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6782 [2026-10-20 14:07:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3255 [2026-10-20 14:07:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8074 [2026-10-20 14:07:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2194 [2026-10-20 14:07:38][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,419 | MFU: 1.8% [2026-10-20 14:07:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8251 [2026-10-20 14:07:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9056 [2026-10-20 14:07:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8547 [2026-10-20 14:07:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7968 [2026-10-20 14:07:43][INFO][training.pretrain_engine] [Ph2|Step 487] loss=5.9015 | ppl=365.60 | aux=0.02512 | lr=3.00e-04 | gnorm=0.902 | tok/s=29,211 | MFU=7.0% | step=143588ms | tokens=0.518B | mem=4.2/32.7GB | cost=$255.94/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 07:50 (137.7h) | Phase3: SKIPPED | SFT: Oct 26 12:35 (142.5h) | DPO (Final): Oct 26 13:10 (143.0h) [2026-10-20 14:07:43][INFO][training.pretrain_engine] Starting global step 488 (accumulating 32 micro-batches)... [2026-10-20 14:07:43][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 14:08:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2430 [2026-10-20 14:08:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0928 [2026-10-20 14:08:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0862 [2026-10-20 14:08:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2670 [2026-10-20 14:08:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7568 [2026-10-20 14:08:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7806 [2026-10-20 14:08:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6547 [2026-10-20 14:08:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9308 [2026-10-20 14:08:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9871 [2026-10-20 14:08:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9245 [2026-10-20 14:08:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8384 [2026-10-20 14:08:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1226 [2026-10-20 14:08:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7216 [2026-10-20 14:08:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9594 [2026-10-20 14:08:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7862 [2026-10-20 14:08:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4010 [2026-10-20 14:09:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0061 [2026-10-20 14:09:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7385 [2026-10-20 14:09:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2872 [2026-10-20 14:09:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9808 [2026-10-20 14:09:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7422 [2026-10-20 14:09:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3493 [2026-10-20 14:09:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6685 [2026-10-20 14:09:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3443 [2026-10-20 14:09:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6072 [2026-10-20 14:09:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6958 [2026-10-20 14:09:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1318 [2026-10-20 14:09:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3653 [2026-10-20 14:10:01][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,555 | MFU: 1.8% [2026-10-20 14:10:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1729 [2026-10-20 14:10:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1704 [2026-10-20 14:10:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3443 [2026-10-20 14:10:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2802 [2026-10-20 14:10:06][INFO][training.pretrain_engine] [Ph2|Step 488] loss=5.8952 | ppl=363.31 | aux=0.02510 | lr=3.00e-04 | gnorm=0.977 | tok/s=29,211 | MFU=7.0% | step=143586ms | tokens=0.519B | mem=4.2/32.7GB | cost=$256.46/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 07:52 (137.7h) | Phase3: SKIPPED | SFT: Oct 26 12:37 (142.5h) | DPO (Final): Oct 26 13:11 (143.0h) [2026-10-20 14:10:09][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step488 | Tokens: 0.519B | Spent: $256.47 | Remaining: $143.53 [2026-10-20 14:10:09][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step488 [2026-10-20 14:10:09][INFO][training.pretrain_engine] Starting global step 489 (accumulating 32 micro-batches)... [2026-10-20 14:10:09][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 14:10:09][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 14:10:09][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 14:10:09][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 14:10:09][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step488 to Phase-Technologies/forge-500m-pretrain (path: phase2_step488)... [2026-10-20 14:10:09][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 14:10:19][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 14:10:19][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step488 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 14:10:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4403 [2026-10-20 14:10:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6748 [2026-10-20 14:10:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6201 [2026-10-20 14:10:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3278 [2026-10-20 14:10:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1428 [2026-10-20 14:10:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2030 [2026-10-20 14:10:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2026 [2026-10-20 14:10:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1508 [2026-10-20 14:10:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3776 [2026-10-20 14:11:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9650 [2026-10-20 14:11:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7161 [2026-10-20 14:11:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6586 [2026-10-20 14:11:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.4342 [2026-10-20 14:11:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9905 [2026-10-20 14:11:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1093 [2026-10-20 14:11:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5388 [2026-10-20 14:11:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2667 [2026-10-20 14:11:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7768 [2026-10-20 14:11:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6768 [2026-10-20 14:11:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1436 [2026-10-20 14:11:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1418 [2026-10-20 14:11:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2500 [2026-10-20 14:11:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2072 [2026-10-20 14:11:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8836 [2026-10-20 14:12:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7083 [2026-10-20 14:12:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3156 [2026-10-20 14:12:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1214 [2026-10-20 14:12:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6105 [2026-10-20 14:12:28][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,570 | MFU: 1.8% [2026-10-20 14:12:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2861 [2026-10-20 14:12:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8037 [2026-10-20 14:12:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0890 [2026-10-20 14:12:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3364 [2026-10-20 14:12:32][INFO][training.pretrain_engine] [Ph2|Step 489] loss=5.9291 | ppl=375.82 | aux=0.02511 | lr=3.00e-04 | gnorm=0.820 | tok/s=29,212 | MFU=7.0% | step=143582ms | tokens=0.520B | mem=4.2/32.7GB | cost=$256.99/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 07:53 (137.7h) | Phase3: SKIPPED | SFT: Oct 26 12:39 (142.4h) | DPO (Final): Oct 26 13:13 (143.0h) [2026-10-20 14:12:32][INFO][training.pretrain_engine] Starting global step 490 (accumulating 32 micro-batches)... [2026-10-20 14:12:32][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 14:12:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6526 [2026-10-20 14:12:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3609 [2026-10-20 14:12:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1454 [2026-10-20 14:12:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9615 [2026-10-20 14:13:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8666 [2026-10-20 14:13:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5739 [2026-10-20 14:13:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3522 [2026-10-20 14:13:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5994 [2026-10-20 14:13:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2960 [2026-10-20 14:13:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1573 [2026-10-20 14:13:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7586 [2026-10-20 14:13:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8918 [2026-10-20 14:13:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6113 [2026-10-20 14:13:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6154 [2026-10-20 14:13:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3599 [2026-10-20 14:13:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8433 [2026-10-20 14:14:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5884 [2026-10-20 14:14:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0678 [2026-10-20 14:14:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9461 [2026-10-20 14:14:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9253 [2026-10-20 14:14:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4201 [2026-10-20 14:14:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0995 [2026-10-20 14:14:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9127 [2026-10-20 14:14:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.6156 [2026-10-20 14:14:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5523 [2026-10-20 14:14:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4853 [2026-10-20 14:14:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9989 [2026-10-20 14:14:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2336 [2026-10-20 14:14:51][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,558 | MFU: 1.8% [2026-10-20 14:14:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1078 [2026-10-20 14:14:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8557 [2026-10-20 14:14:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6476 [2026-10-20 14:14:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6690 [2026-10-20 14:14:55][INFO][training.pretrain_engine] [Ph2|Step 490] loss=5.9375 | ppl=378.97 | aux=0.02510 | lr=3.00e-04 | gnorm=0.855 | tok/s=29,212 | MFU=7.0% | step=143580ms | tokens=0.521B | mem=4.2/32.7GB | cost=$257.52/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 07:55 (137.7h) | Phase3: SKIPPED | SFT: Oct 26 12:40 (142.4h) | DPO (Final): Oct 26 13:15 (143.0h) [2026-10-20 14:14:55][INFO][training.pretrain_engine] Starting global step 491 (accumulating 32 micro-batches)... [2026-10-20 14:14:55][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 14:15:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3908 [2026-10-20 14:15:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0676 [2026-10-20 14:15:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2847 [2026-10-20 14:15:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6604 [2026-10-20 14:15:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9624 [2026-10-20 14:15:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2152 [2026-10-20 14:15:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1464 [2026-10-20 14:15:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0336 [2026-10-20 14:15:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9822 [2026-10-20 14:15:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0907 [2026-10-20 14:15:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8339 [2026-10-20 14:15:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4805 [2026-10-20 14:16:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8416 [2026-10-20 14:16:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2869 [2026-10-20 14:16:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3113 [2026-10-20 14:16:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9509 [2026-10-20 14:16:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0502 [2026-10-20 14:16:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8403 [2026-10-20 14:16:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6878 [2026-10-20 14:16:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0166 [2026-10-20 14:16:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9189 [2026-10-20 14:16:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7701 [2026-10-20 14:16:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1243 [2026-10-20 14:16:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5000 [2026-10-20 14:17:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9208 [2026-10-20 14:17:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8040 [2026-10-20 14:17:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1398 [2026-10-20 14:17:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1063 [2026-10-20 14:17:14][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,568 | MFU: 1.8% [2026-10-20 14:17:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0365 [2026-10-20 14:17:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4589 [2026-10-20 14:17:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2246 [2026-10-20 14:17:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0649 [2026-10-20 14:17:19][INFO][training.pretrain_engine] [Ph2|Step 491] loss=5.9692 | ppl=391.20 | aux=0.02510 | lr=3.00e-04 | gnorm=0.902 | tok/s=29,213 | MFU=7.0% | step=143576ms | tokens=0.522B | mem=4.2/32.7GB | cost=$258.04/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 07:57 (137.7h) | Phase3: SKIPPED | SFT: Oct 26 12:42 (142.4h) | DPO (Final): Oct 26 13:16 (143.0h) [2026-10-20 14:17:19][INFO][training.pretrain_engine] Starting global step 492 (accumulating 32 micro-batches)... [2026-10-20 14:17:19][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 14:17:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7491 [2026-10-20 14:17:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3124 [2026-10-20 14:17:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9177 [2026-10-20 14:17:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9829 [2026-10-20 14:17:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1004 [2026-10-20 14:17:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0106 [2026-10-20 14:17:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9781 [2026-10-20 14:17:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1308 [2026-10-20 14:18:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0843 [2026-10-20 14:18:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8331 [2026-10-20 14:18:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=7.1835 [2026-10-20 14:18:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2843 [2026-10-20 14:18:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4843 [2026-10-20 14:18:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5006 [2026-10-20 14:18:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9455 [2026-10-20 14:18:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8671 [2026-10-20 14:18:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7095 [2026-10-20 14:18:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5006 [2026-10-20 14:18:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5712 [2026-10-20 14:18:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4902 [2026-10-20 14:19:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0715 [2026-10-20 14:19:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5506 [2026-10-20 14:19:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8559 [2026-10-20 14:19:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9117 [2026-10-20 14:19:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1008 [2026-10-20 14:19:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0712 [2026-10-20 14:19:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9857 [2026-10-20 14:19:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9475 [2026-10-20 14:19:37][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,579 | MFU: 1.8% [2026-10-20 14:19:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5432 [2026-10-20 14:19:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1112 [2026-10-20 14:19:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9884 [2026-10-20 14:19:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9188 [2026-10-20 14:19:42][INFO][training.pretrain_engine] [Ph2|Step 492] loss=6.0053 | ppl=405.56 | aux=0.02511 | lr=3.00e-04 | gnorm=0.801 | tok/s=29,215 | MFU=7.0% | step=143566ms | tokens=0.523B | mem=4.2/32.7GB | cost=$258.56/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 07:58 (137.6h) | Phase3: SKIPPED | SFT: Oct 26 12:43 (142.4h) | DPO (Final): Oct 26 13:17 (143.0h) [2026-10-20 14:19:42][INFO][training.pretrain_engine] Starting global step 493 (accumulating 32 micro-batches)... [2026-10-20 14:19:42][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 14:19:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7097 [2026-10-20 14:19:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9391 [2026-10-20 14:19:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8494 [2026-10-20 14:19:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6755 [2026-10-20 14:20:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2293 [2026-10-20 14:20:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2101 [2026-10-20 14:20:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8640 [2026-10-20 14:20:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2449 [2026-10-20 14:20:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1699 [2026-10-20 14:20:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9990 [2026-10-20 14:20:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3492 [2026-10-20 14:20:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8705 [2026-10-20 14:20:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3759 [2026-10-20 14:20:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7102 [2026-10-20 14:20:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8186 [2026-10-20 14:20:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6685 [2026-10-20 14:21:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1205 [2026-10-20 14:21:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8912 [2026-10-20 14:21:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0820 [2026-10-20 14:21:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3562 [2026-10-20 14:21:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6250 [2026-10-20 14:21:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8087 [2026-10-20 14:21:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6296 [2026-10-20 14:21:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7415 [2026-10-20 14:21:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0398 [2026-10-20 14:21:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6856 [2026-10-20 14:21:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8441 [2026-10-20 14:21:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1485 [2026-10-20 14:22:00][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,567 | MFU: 1.8% [2026-10-20 14:22:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8231 [2026-10-20 14:22:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7384 [2026-10-20 14:22:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2005 [2026-10-20 14:22:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8037 [2026-10-20 14:22:05][INFO][training.pretrain_engine] [Ph2|Step 493] loss=5.8454 | ppl=345.65 | aux=0.02509 | lr=3.00e-04 | gnorm=0.824 | tok/s=29,217 | MFU=7.0% | step=143557ms | tokens=0.524B | mem=4.2/32.7GB | cost=$259.09/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 07:59 (137.6h) | Phase3: SKIPPED | SFT: Oct 26 12:44 (142.4h) | DPO (Final): Oct 26 13:19 (143.0h) [2026-10-20 14:22:05][INFO][training.pretrain_engine] Starting global step 494 (accumulating 32 micro-batches)... [2026-10-20 14:22:05][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 14:22:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4104 [2026-10-20 14:22:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7712 [2026-10-20 14:22:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5277 [2026-10-20 14:22:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5844 [2026-10-20 14:22:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4252 [2026-10-20 14:22:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7566 [2026-10-20 14:22:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9808 [2026-10-20 14:22:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5407 [2026-10-20 14:22:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6298 [2026-10-20 14:22:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1808 [2026-10-20 14:22:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5286 [2026-10-20 14:22:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4545 [2026-10-20 14:23:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9291 [2026-10-20 14:23:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9667 [2026-10-20 14:23:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9278 [2026-10-20 14:23:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0415 [2026-10-20 14:23:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3435 [2026-10-20 14:23:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5158 [2026-10-20 14:23:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6831 [2026-10-20 14:23:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4745 [2026-10-20 14:23:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2024 [2026-10-20 14:23:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8751 [2026-10-20 14:23:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0513 [2026-10-20 14:23:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5902 [2026-10-20 14:24:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7824 [2026-10-20 14:24:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6931 [2026-10-20 14:24:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4970 [2026-10-20 14:24:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0261 [2026-10-20 14:24:23][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,584 | MFU: 1.8% [2026-10-20 14:24:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9999 [2026-10-20 14:24:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0884 [2026-10-20 14:24:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2969 [2026-10-20 14:24:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4527 [2026-10-20 14:24:28][INFO][training.pretrain_engine] [Ph2|Step 494] loss=5.9999 | ppl=403.40 | aux=0.02509 | lr=3.00e-04 | gnorm=0.699 | tok/s=29,218 | MFU=7.0% | step=143552ms | tokens=0.525B | mem=4.2/32.7GB | cost=$259.61/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 08:01 (137.6h) | Phase3: SKIPPED | SFT: Oct 26 12:46 (142.4h) | DPO (Final): Oct 26 13:20 (142.9h) [2026-10-20 14:24:28][INFO][training.pretrain_engine] Starting global step 495 (accumulating 32 micro-batches)... [2026-10-20 14:24:28][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 14:24:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5502 [2026-10-20 14:24:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5634 [2026-10-20 14:24:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9233 [2026-10-20 14:24:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2867 [2026-10-20 14:25:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7126 [2026-10-20 14:25:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9724 [2026-10-20 14:25:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1964 [2026-10-20 14:25:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5741 [2026-10-20 14:25:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7997 [2026-10-20 14:25:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5466 [2026-10-20 14:25:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8734 [2026-10-20 14:25:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1109 [2026-10-20 14:25:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1114 [2026-10-20 14:25:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4877 [2026-10-20 14:25:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7202 [2026-10-20 14:25:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4250 [2026-10-20 14:25:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6294 [2026-10-20 14:25:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0654 [2026-10-20 14:25:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5512 [2026-10-20 14:25:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4290 [2026-10-20 14:26:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0810 [2026-10-20 14:26:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3141 [2026-10-20 14:26:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0859 [2026-10-20 14:26:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3410 [2026-10-20 14:26:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3972 [2026-10-20 14:26:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8051 [2026-10-20 14:26:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2095 [2026-10-20 14:26:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7140 [2026-10-20 14:26:46][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,562 | MFU: 1.8% [2026-10-20 14:26:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8723 [2026-10-20 14:26:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2226 [2026-10-20 14:26:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0834 [2026-10-20 14:26:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9702 [2026-10-20 14:26:51][INFO][training.pretrain_engine] [Ph2|Step 495] loss=5.7781 | ppl=323.14 | aux=0.02510 | lr=3.00e-04 | gnorm=0.832 | tok/s=29,219 | MFU=7.0% | step=143547ms | tokens=0.526B | mem=4.2/32.7GB | cost=$260.13/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 08:02 (137.6h) | Phase3: SKIPPED | SFT: Oct 26 12:47 (142.4h) | DPO (Final): Oct 26 13:22 (142.9h) [2026-10-20 14:26:51][INFO][training.pretrain_engine] Starting global step 496 (accumulating 32 micro-batches)... [2026-10-20 14:26:51][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 14:27:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2010 [2026-10-20 14:27:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6541 [2026-10-20 14:27:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2563 [2026-10-20 14:27:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3654 [2026-10-20 14:27:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9029 [2026-10-20 14:27:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4186 [2026-10-20 14:27:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9107 [2026-10-20 14:27:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7242 [2026-10-20 14:27:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7000 [2026-10-20 14:27:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7076 [2026-10-20 14:27:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0162 [2026-10-20 14:27:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5801 [2026-10-20 14:28:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1931 [2026-10-20 14:28:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8139 [2026-10-20 14:28:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0307 [2026-10-20 14:28:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1911 [2026-10-20 14:28:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2909 [2026-10-20 14:28:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0050 [2026-10-20 14:28:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0297 [2026-10-20 14:28:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0395 [2026-10-20 14:28:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4836 [2026-10-20 14:28:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3390 [2026-10-20 14:28:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2692 [2026-10-20 14:28:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2850 [2026-10-20 14:28:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8438 [2026-10-20 14:28:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8296 [2026-10-20 14:28:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6453 [2026-10-20 14:28:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8215 [2026-10-20 14:29:09][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,563 | MFU: 1.8% [2026-10-20 14:29:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1333 [2026-10-20 14:29:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9568 [2026-10-20 14:29:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1364 [2026-10-20 14:29:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7485 [2026-10-20 14:29:14][INFO][training.pretrain_engine] [Ph2|Step 496] loss=5.8662 | ppl=352.89 | aux=0.02510 | lr=3.00e-04 | gnorm=0.809 | tok/s=29,220 | MFU=7.0% | step=143542ms | tokens=0.527B | mem=4.2/32.7GB | cost=$260.66/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 08:04 (137.6h) | Phase3: SKIPPED | SFT: Oct 26 12:49 (142.3h) | DPO (Final): Oct 26 13:23 (142.9h) [2026-10-20 14:29:14][INFO][training.pretrain_engine] Starting global step 497 (accumulating 32 micro-batches)... [2026-10-20 14:29:14][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 14:29:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2870 [2026-10-20 14:29:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2926 [2026-10-20 14:29:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2113 [2026-10-20 14:29:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2683 [2026-10-20 14:29:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9887 [2026-10-20 14:29:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1598 [2026-10-20 14:29:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3955 [2026-10-20 14:29:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2743 [2026-10-20 14:30:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1749 [2026-10-20 14:30:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8936 [2026-10-20 14:30:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0858 [2026-10-20 14:30:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9018 [2026-10-20 14:30:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5402 [2026-10-20 14:30:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1282 [2026-10-20 14:30:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6356 [2026-10-20 14:30:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7186 [2026-10-20 14:30:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4959 [2026-10-20 14:30:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6687 [2026-10-20 14:30:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7748 [2026-10-20 14:30:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5194 [2026-10-20 14:31:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5225 [2026-10-20 14:31:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7957 [2026-10-20 14:31:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9751 [2026-10-20 14:31:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8910 [2026-10-20 14:31:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4760 [2026-10-20 14:31:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5994 [2026-10-20 14:31:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7136 [2026-10-20 14:31:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0333 [2026-10-20 14:31:33][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,560 | MFU: 1.8% [2026-10-20 14:31:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6868 [2026-10-20 14:31:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8342 [2026-10-20 14:31:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9068 [2026-10-20 14:31:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8358 [2026-10-20 14:31:37][INFO][training.pretrain_engine] [Ph2|Step 497] loss=5.9511 | ppl=384.16 | aux=0.02510 | lr=3.00e-04 | gnorm=0.793 | tok/s=29,219 | MFU=7.0% | step=143546ms | tokens=0.528B | mem=4.2/32.7GB | cost=$261.18/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 08:06 (137.6h) | Phase3: SKIPPED | SFT: Oct 26 12:51 (142.3h) | DPO (Final): Oct 26 13:25 (142.9h) [2026-10-20 14:31:40][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step497 | Tokens: 0.528B | Spent: $261.19 | Remaining: $138.81 [2026-10-20 14:31:40][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step497 [2026-10-20 14:31:40][INFO][training.pretrain_engine] Starting global step 498 (accumulating 32 micro-batches)... [2026-10-20 14:31:40][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 14:31:40][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 14:31:40][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 14:31:41][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 14:31:41][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step497 to Phase-Technologies/forge-500m-pretrain (path: phase2_step497)... [2026-10-20 14:31:41][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 14:31:51][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 14:31:51][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step497 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 14:31:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6606 [2026-10-20 14:31:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4020 [2026-10-20 14:31:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7527 [2026-10-20 14:31:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7488 [2026-10-20 14:32:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5216 [2026-10-20 14:32:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3008 [2026-10-20 14:32:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2040 [2026-10-20 14:32:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1684 [2026-10-20 14:32:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4435 [2026-10-20 14:32:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7222 [2026-10-20 14:32:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7931 [2026-10-20 14:32:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7094 [2026-10-20 14:32:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1287 [2026-10-20 14:32:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3594 [2026-10-20 14:32:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1476 [2026-10-20 14:32:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2102 [2026-10-20 14:33:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2459 [2026-10-20 14:33:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8605 [2026-10-20 14:33:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8664 [2026-10-20 14:33:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8452 [2026-10-20 14:33:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5139 [2026-10-20 14:33:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6982 [2026-10-20 14:33:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7201 [2026-10-20 14:33:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1400 [2026-10-20 14:33:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8664 [2026-10-20 14:33:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5574 [2026-10-20 14:33:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4193 [2026-10-20 14:33:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7165 [2026-10-20 14:34:00][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,534 | MFU: 1.8% [2026-10-20 14:34:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6423 [2026-10-20 14:34:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9896 [2026-10-20 14:34:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8400 [2026-10-20 14:34:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0757 [2026-10-20 14:34:04][INFO][training.pretrain_engine] [Ph2|Step 498] loss=5.8600 | ppl=350.72 | aux=0.02511 | lr=3.00e-04 | gnorm=0.902 | tok/s=29,219 | MFU=7.0% | step=143546ms | tokens=0.530B | mem=4.2/32.7GB | cost=$261.72/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 08:08 (137.6h) | Phase3: SKIPPED | SFT: Oct 26 12:53 (142.3h) | DPO (Final): Oct 26 13:27 (142.9h) [2026-10-20 14:34:04][INFO][training.pretrain_engine] Starting global step 499 (accumulating 32 micro-batches)... [2026-10-20 14:34:04][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 14:34:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2550 [2026-10-20 14:34:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9436 [2026-10-20 14:34:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4212 [2026-10-20 14:34:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6244 [2026-10-20 14:34:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.7814 [2026-10-20 14:34:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2253 [2026-10-20 14:34:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3602 [2026-10-20 14:34:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4320 [2026-10-20 14:34:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3557 [2026-10-20 14:34:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2853 [2026-10-20 14:34:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1142 [2026-10-20 14:34:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1106 [2026-10-20 14:35:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6605 [2026-10-20 14:35:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0112 [2026-10-20 14:35:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9715 [2026-10-20 14:35:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8816 [2026-10-20 14:35:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4766 [2026-10-20 14:35:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9483 [2026-10-20 14:35:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1200 [2026-10-20 14:35:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0750 [2026-10-20 14:35:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9877 [2026-10-20 14:35:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1316 [2026-10-20 14:35:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2538 [2026-10-20 14:35:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0234 [2026-10-20 14:36:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7462 [2026-10-20 14:36:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8458 [2026-10-20 14:36:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0241 [2026-10-20 14:36:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8475 [2026-10-20 14:36:23][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,568 | MFU: 1.8% [2026-10-20 14:36:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8716 [2026-10-20 14:36:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8531 [2026-10-20 14:36:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3501 [2026-10-20 14:36:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.5123 [2026-10-20 14:36:27][INFO][training.pretrain_engine] [Ph2|Step 499] loss=5.9621 | ppl=388.41 | aux=0.02510 | lr=3.00e-04 | gnorm=1.156 | tok/s=29,220 | MFU=7.0% | step=143541ms | tokens=0.531B | mem=4.2/32.7GB | cost=$262.24/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 08:09 (137.6h) | Phase3: SKIPPED | SFT: Oct 26 12:54 (142.3h) | DPO (Final): Oct 26 13:28 (142.9h) [2026-10-20 14:36:27][INFO][training.pretrain_engine] Starting global step 500 (accumulating 32 micro-batches)... [2026-10-20 14:36:27][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 14:36:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8937 [2026-10-20 14:36:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6806 [2026-10-20 14:36:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1322 [2026-10-20 14:36:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5840 [2026-10-20 14:37:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2545 [2026-10-20 14:37:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5504 [2026-10-20 14:37:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8472 [2026-10-20 14:37:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0155 [2026-10-20 14:37:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8845 [2026-10-20 14:37:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8567 [2026-10-20 14:37:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2425 [2026-10-20 14:37:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6121 [2026-10-20 14:37:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9225 [2026-10-20 14:37:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1991 [2026-10-20 14:37:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2229 [2026-10-20 14:37:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5971 [2026-10-20 14:37:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4270 [2026-10-20 14:37:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7669 [2026-10-20 14:37:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5341 [2026-10-20 14:37:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8891 [2026-10-20 14:38:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9463 [2026-10-20 14:38:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8279 [2026-10-20 14:38:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9660 [2026-10-20 14:38:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0415 [2026-10-20 14:38:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8306 [2026-10-20 14:38:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8655 [2026-10-20 14:38:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9143 [2026-10-20 14:38:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4877 [2026-10-20 14:38:46][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,562 | MFU: 1.8% [2026-10-20 14:38:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9311 [2026-10-20 14:38:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3228 [2026-10-20 14:38:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0528 [2026-10-20 14:38:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0277 [2026-10-20 14:38:51][INFO][training.pretrain_engine] [Ph2|Step 500] loss=5.9043 | ppl=366.62 | aux=0.02512 | lr=3.00e-04 | gnorm=0.934 | tok/s=29,221 | MFU=7.0% | step=143535ms | tokens=0.532B | mem=4.2/32.7GB | cost=$262.76/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 08:11 (137.5h) | Phase3: SKIPPED | SFT: Oct 26 12:56 (142.3h) | DPO (Final): Oct 26 13:30 (142.9h) [2026-10-20 14:38:51][INFO][training.pretrain_engine] Starting global step 501 (accumulating 32 micro-batches)... [2026-10-20 14:38:51][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 14:39:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7536 [2026-10-20 14:39:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5424 [2026-10-20 14:39:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0949 [2026-10-20 14:39:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3230 [2026-10-20 14:39:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3074 [2026-10-20 14:39:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2472 [2026-10-20 14:39:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1323 [2026-10-20 14:39:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0695 [2026-10-20 14:39:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9599 [2026-10-20 14:39:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0256 [2026-10-20 14:39:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4493 [2026-10-20 14:39:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1248 [2026-10-20 14:40:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7962 [2026-10-20 14:40:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4927 [2026-10-20 14:40:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6492 [2026-10-20 14:40:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6395 [2026-10-20 14:40:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3261 [2026-10-20 14:40:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3315 [2026-10-20 14:40:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7544 [2026-10-20 14:40:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2499 [2026-10-20 14:40:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9689 [2026-10-20 14:40:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7566 [2026-10-20 14:40:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9269 [2026-10-20 14:40:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9156 [2026-10-20 14:40:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6288 [2026-10-20 14:40:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1199 [2026-10-20 14:40:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6046 [2026-10-20 14:40:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2932 [2026-10-20 14:41:09][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,591 | MFU: 1.8% [2026-10-20 14:41:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3971 [2026-10-20 14:41:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8380 [2026-10-20 14:41:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7766 [2026-10-20 14:41:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1138 [2026-10-20 14:41:14][INFO][training.pretrain_engine] [Ph2|Step 501] loss=5.9672 | ppl=390.42 | aux=0.02511 | lr=3.00e-04 | gnorm=0.781 | tok/s=29,223 | MFU=7.0% | step=143527ms | tokens=0.533B | mem=4.2/32.7GB | cost=$263.29/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 08:12 (137.5h) | Phase3: SKIPPED | SFT: Oct 26 12:57 (142.3h) | DPO (Final): Oct 26 13:31 (142.8h) [2026-10-20 14:41:14][INFO][training.pretrain_engine] Starting global step 502 (accumulating 32 micro-batches)... [2026-10-20 14:41:14][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 14:41:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5590 [2026-10-20 14:41:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5053 [2026-10-20 14:41:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3276 [2026-10-20 14:41:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4122 [2026-10-20 14:41:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5462 [2026-10-20 14:41:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6032 [2026-10-20 14:41:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8687 [2026-10-20 14:41:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7807 [2026-10-20 14:42:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5318 [2026-10-20 14:42:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2217 [2026-10-20 14:42:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0411 [2026-10-20 14:42:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1218 [2026-10-20 14:42:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0351 [2026-10-20 14:42:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9287 [2026-10-20 14:42:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7636 [2026-10-20 14:42:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4241 [2026-10-20 14:42:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6834 [2026-10-20 14:42:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0646 [2026-10-20 14:42:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0231 [2026-10-20 14:42:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1299 [2026-10-20 14:43:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7761 [2026-10-20 14:43:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0205 [2026-10-20 14:43:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0705 [2026-10-20 14:43:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9041 [2026-10-20 14:43:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2557 [2026-10-20 14:43:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4739 [2026-10-20 14:43:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6620 [2026-10-20 14:43:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7428 [2026-10-20 14:43:32][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,566 | MFU: 1.8% [2026-10-20 14:43:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0289 [2026-10-20 14:43:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5502 [2026-10-20 14:43:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8554 [2026-10-20 14:43:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1283 [2026-10-20 14:43:37][INFO][training.pretrain_engine] [Ph2|Step 502] loss=5.8945 | ppl=363.04 | aux=0.02510 | lr=3.00e-04 | gnorm=1.055 | tok/s=29,224 | MFU=7.0% | step=143525ms | tokens=0.534B | mem=4.2/32.7GB | cost=$263.81/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 08:13 (137.5h) | Phase3: SKIPPED | SFT: Oct 26 12:59 (142.3h) | DPO (Final): Oct 26 13:33 (142.8h) [2026-10-20 14:43:37][INFO][training.pretrain_engine] Starting global step 503 (accumulating 32 micro-batches)... [2026-10-20 14:43:37][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 14:43:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7481 [2026-10-20 14:43:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3536 [2026-10-20 14:43:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6956 [2026-10-20 14:43:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7469 [2026-10-20 14:44:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1997 [2026-10-20 14:44:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5837 [2026-10-20 14:44:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4666 [2026-10-20 14:44:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8937 [2026-10-20 14:44:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6611 [2026-10-20 14:44:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7179 [2026-10-20 14:44:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9606 [2026-10-20 14:44:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7344 [2026-10-20 14:44:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7618 [2026-10-20 14:44:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7665 [2026-10-20 14:44:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3894 [2026-10-20 14:44:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8405 [2026-10-20 14:45:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8996 [2026-10-20 14:45:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6983 [2026-10-20 14:45:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0459 [2026-10-20 14:45:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2293 [2026-10-20 14:45:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0618 [2026-10-20 14:45:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9536 [2026-10-20 14:45:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0026 [2026-10-20 14:45:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2394 [2026-10-20 14:45:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1056 [2026-10-20 14:45:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4792 [2026-10-20 14:45:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2708 [2026-10-20 14:45:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5867 [2026-10-20 14:45:55][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,558 | MFU: 1.8% [2026-10-20 14:46:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3913 [2026-10-20 14:46:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9037 [2026-10-20 14:46:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8775 [2026-10-20 14:46:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3314 [2026-10-20 14:46:00][INFO][training.pretrain_engine] [Ph2|Step 503] loss=5.9579 | ppl=386.81 | aux=0.02509 | lr=3.00e-04 | gnorm=0.996 | tok/s=29,224 | MFU=7.0% | step=143521ms | tokens=0.535B | mem=4.2/32.7GB | cost=$264.33/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 08:15 (137.5h) | Phase3: SKIPPED | SFT: Oct 26 13:00 (142.2h) | DPO (Final): Oct 26 13:34 (142.8h) [2026-10-20 14:46:00][INFO][training.pretrain_engine] Starting global step 504 (accumulating 32 micro-batches)... [2026-10-20 14:46:00][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 14:46:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7270 [2026-10-20 14:46:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1744 [2026-10-20 14:46:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7679 [2026-10-20 14:46:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4685 [2026-10-20 14:46:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7350 [2026-10-20 14:46:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6163 [2026-10-20 14:46:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9393 [2026-10-20 14:46:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9325 [2026-10-20 14:46:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0306 [2026-10-20 14:46:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3233 [2026-10-20 14:46:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5759 [2026-10-20 14:46:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2932 [2026-10-20 14:47:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7036 [2026-10-20 14:47:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4228 [2026-10-20 14:47:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6712 [2026-10-20 14:47:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7284 [2026-10-20 14:47:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9219 [2026-10-20 14:47:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0951 [2026-10-20 14:47:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1877 [2026-10-20 14:47:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1095 [2026-10-20 14:47:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2756 [2026-10-20 14:47:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6889 [2026-10-20 14:47:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9133 [2026-10-20 14:47:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3322 [2026-10-20 14:48:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8617 [2026-10-20 14:48:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8920 [2026-10-20 14:48:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7999 [2026-10-20 14:48:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4419 [2026-10-20 14:48:19][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,566 | MFU: 1.8% [2026-10-20 14:48:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4582 [2026-10-20 14:48:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7245 [2026-10-20 14:48:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9057 [2026-10-20 14:48:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0485 [2026-10-20 14:48:23][INFO][training.pretrain_engine] [Ph2|Step 504] loss=5.9531 | ppl=384.94 | aux=0.02508 | lr=3.00e-04 | gnorm=1.172 | tok/s=29,225 | MFU=7.0% | step=143516ms | tokens=0.536B | mem=4.2/32.7GB | cost=$264.86/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 08:17 (137.5h) | Phase3: SKIPPED | SFT: Oct 26 13:02 (142.2h) | DPO (Final): Oct 26 13:36 (142.8h) [2026-10-20 14:48:23][INFO][training.pretrain_engine] Starting global step 505 (accumulating 32 micro-batches)... [2026-10-20 14:48:23][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 14:48:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8552 [2026-10-20 14:48:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7618 [2026-10-20 14:48:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5499 [2026-10-20 14:48:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7701 [2026-10-20 14:48:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0682 [2026-10-20 14:48:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9099 [2026-10-20 14:48:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3107 [2026-10-20 14:48:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5567 [2026-10-20 14:49:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8677 [2026-10-20 14:49:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3964 [2026-10-20 14:49:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5923 [2026-10-20 14:49:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6324 [2026-10-20 14:49:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3568 [2026-10-20 14:49:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6391 [2026-10-20 14:49:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4975 [2026-10-20 14:49:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1396 [2026-10-20 14:49:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0966 [2026-10-20 14:49:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8696 [2026-10-20 14:49:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4052 [2026-10-20 14:49:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6066 [2026-10-20 14:50:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0471 [2026-10-20 14:50:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0749 [2026-10-20 14:50:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8437 [2026-10-20 14:50:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1064 [2026-10-20 14:50:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9408 [2026-10-20 14:50:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9027 [2026-10-20 14:50:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2607 [2026-10-20 14:50:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6429 [2026-10-20 14:50:41][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,580 | MFU: 1.8% [2026-10-20 14:50:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8400 [2026-10-20 14:50:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5053 [2026-10-20 14:50:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0805 [2026-10-20 14:50:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4688 [2026-10-20 14:50:46][INFO][training.pretrain_engine] [Ph2|Step 505] loss=5.8351 | ppl=342.09 | aux=0.02509 | lr=3.00e-04 | gnorm=0.855 | tok/s=29,226 | MFU=7.0% | step=143513ms | tokens=0.537B | mem=4.2/32.7GB | cost=$265.38/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 08:18 (137.5h) | Phase3: SKIPPED | SFT: Oct 26 13:03 (142.2h) | DPO (Final): Oct 26 13:38 (142.8h) [2026-10-20 14:50:46][INFO][training.pretrain_engine] Starting global step 506 (accumulating 32 micro-batches)... [2026-10-20 14:50:46][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 14:51:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5936 [2026-10-20 14:51:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3289 [2026-10-20 14:51:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7435 [2026-10-20 14:51:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0953 [2026-10-20 14:51:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4501 [2026-10-20 14:51:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4122 [2026-10-20 14:51:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8020 [2026-10-20 14:51:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8324 [2026-10-20 14:51:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3766 [2026-10-20 14:51:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7017 [2026-10-20 14:51:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1513 [2026-10-20 14:51:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0787 [2026-10-20 14:51:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9281 [2026-10-20 14:51:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9245 [2026-10-20 14:51:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1152 [2026-10-20 14:51:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1135 [2026-10-20 14:52:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9881 [2026-10-20 14:52:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0249 [2026-10-20 14:52:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9167 [2026-10-20 14:52:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3882 [2026-10-20 14:52:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8451 [2026-10-20 14:52:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7010 [2026-10-20 14:52:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3700 [2026-10-20 14:52:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5997 [2026-10-20 14:52:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5561 [2026-10-20 14:52:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9413 [2026-10-20 14:52:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9659 [2026-10-20 14:52:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8718 [2026-10-20 14:53:05][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,530 | MFU: 1.8% [2026-10-20 14:53:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5898 [2026-10-20 14:53:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7521 [2026-10-20 14:53:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1270 [2026-10-20 14:53:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7900 [2026-10-20 14:53:10][INFO][training.pretrain_engine] [Ph2|Step 506] loss=5.8225 | ppl=337.81 | aux=0.02510 | lr=3.00e-04 | gnorm=1.039 | tok/s=29,226 | MFU=7.0% | step=143514ms | tokens=0.538B | mem=4.2/32.7GB | cost=$265.91/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 08:20 (137.5h) | Phase3: SKIPPED | SFT: Oct 26 13:05 (142.2h) | DPO (Final): Oct 26 13:39 (142.8h) [2026-10-20 14:53:13][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step506 | Tokens: 0.538B | Spent: $265.92 | Remaining: $134.08 [2026-10-20 14:53:13][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step506 [2026-10-20 14:53:13][INFO][training.pretrain_engine] Starting global step 507 (accumulating 32 micro-batches)... [2026-10-20 14:53:13][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 14:53:13][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 14:53:13][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 14:53:13][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 14:53:13][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step506 to Phase-Technologies/forge-500m-pretrain (path: phase2_step506)... [2026-10-20 14:53:13][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 14:53:22][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 14:53:22][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step506 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 14:53:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8716 [2026-10-20 14:53:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3154 [2026-10-20 14:53:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6131 [2026-10-20 14:53:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9408 [2026-10-20 14:53:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9036 [2026-10-20 14:53:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8315 [2026-10-20 14:53:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8076 [2026-10-20 14:53:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0649 [2026-10-20 14:54:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5880 [2026-10-20 14:54:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5874 [2026-10-20 14:54:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2097 [2026-10-20 14:54:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5736 [2026-10-20 14:54:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0338 [2026-10-20 14:54:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9934 [2026-10-20 14:54:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8385 [2026-10-20 14:54:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2316 [2026-10-20 14:54:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1354 [2026-10-20 14:54:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9751 [2026-10-20 14:54:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4579 [2026-10-20 14:54:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6973 [2026-10-20 14:54:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7192 [2026-10-20 14:54:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7919 [2026-10-20 14:54:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9666 [2026-10-20 14:55:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7905 [2026-10-20 14:55:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9672 [2026-10-20 14:55:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3197 [2026-10-20 14:55:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0088 [2026-10-20 14:55:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0059 [2026-10-20 14:55:32][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,563 | MFU: 1.8% [2026-10-20 14:55:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4185 [2026-10-20 14:55:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8371 [2026-10-20 14:55:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0597 [2026-10-20 14:55:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1267 [2026-10-20 14:55:36][INFO][training.pretrain_engine] [Ph2|Step 507] loss=5.8220 | ppl=337.65 | aux=0.02510 | lr=3.00e-04 | gnorm=0.941 | tok/s=29,228 | MFU=7.0% | step=143504ms | tokens=0.539B | mem=4.2/32.7GB | cost=$266.44/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 08:21 (137.4h) | Phase3: SKIPPED | SFT: Oct 26 13:06 (142.2h) | DPO (Final): Oct 26 13:41 (142.8h) [2026-10-20 14:55:36][INFO][training.pretrain_engine] Starting global step 508 (accumulating 32 micro-batches)... [2026-10-20 14:55:36][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 14:55:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8726 [2026-10-20 14:55:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1163 [2026-10-20 14:55:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2040 [2026-10-20 14:55:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5451 [2026-10-20 14:56:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6445 [2026-10-20 14:56:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6430 [2026-10-20 14:56:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8003 [2026-10-20 14:56:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3943 [2026-10-20 14:56:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6420 [2026-10-20 14:56:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6879 [2026-10-20 14:56:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5285 [2026-10-20 14:56:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2643 [2026-10-20 14:56:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5253 [2026-10-20 14:56:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2457 [2026-10-20 14:56:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4776 [2026-10-20 14:56:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4010 [2026-10-20 14:57:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2749 [2026-10-20 14:57:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7599 [2026-10-20 14:57:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9214 [2026-10-20 14:57:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5444 [2026-10-20 14:57:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0692 [2026-10-20 14:57:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0072 [2026-10-20 14:57:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9515 [2026-10-20 14:57:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1356 [2026-10-20 14:57:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1456 [2026-10-20 14:57:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1488 [2026-10-20 14:57:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6316 [2026-10-20 14:57:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2491 [2026-10-20 14:57:55][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,581 | MFU: 1.8% [2026-10-20 14:58:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1433 [2026-10-20 14:58:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9544 [2026-10-20 14:58:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2201 [2026-10-20 14:58:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0137 [2026-10-20 14:58:00][INFO][training.pretrain_engine] [Ph2|Step 508] loss=5.8751 | ppl=356.07 | aux=0.02508 | lr=3.00e-04 | gnorm=1.328 | tok/s=29,230 | MFU=7.0% | step=143495ms | tokens=0.540B | mem=4.2/32.7GB | cost=$266.96/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 08:23 (137.4h) | Phase3: SKIPPED | SFT: Oct 26 13:08 (142.2h) | DPO (Final): Oct 26 13:42 (142.7h) [2026-10-20 14:58:00][INFO][training.pretrain_engine] Starting global step 509 (accumulating 32 micro-batches)... [2026-10-20 14:58:00][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 14:58:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8667 [2026-10-20 14:58:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9397 [2026-10-20 14:58:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4236 [2026-10-20 14:58:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1782 [2026-10-20 14:58:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8895 [2026-10-20 14:58:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7420 [2026-10-20 14:58:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5433 [2026-10-20 14:58:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0852 [2026-10-20 14:58:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6124 [2026-10-20 14:58:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5841 [2026-10-20 14:58:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7155 [2026-10-20 14:58:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7544 [2026-10-20 14:59:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6655 [2026-10-20 14:59:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7047 [2026-10-20 14:59:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0176 [2026-10-20 14:59:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0086 [2026-10-20 14:59:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1261 [2026-10-20 14:59:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7141 [2026-10-20 14:59:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9427 [2026-10-20 14:59:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7603 [2026-10-20 14:59:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7802 [2026-10-20 14:59:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0313 [2026-10-20 14:59:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9207 [2026-10-20 14:59:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8663 [2026-10-20 15:00:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9373 [2026-10-20 15:00:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7262 [2026-10-20 15:00:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7585 [2026-10-20 15:00:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9211 [2026-10-20 15:00:18][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,582 | MFU: 1.8% [2026-10-20 15:00:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2877 [2026-10-20 15:00:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9017 [2026-10-20 15:00:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1535 [2026-10-20 15:00:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1794 [2026-10-20 15:00:22][INFO][training.pretrain_engine] [Ph2|Step 509] loss=5.9334 | ppl=377.44 | aux=0.02508 | lr=3.00e-04 | gnorm=0.621 | tok/s=29,231 | MFU=7.0% | step=143487ms | tokens=0.541B | mem=4.2/32.7GB | cost=$267.49/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 08:24 (137.4h) | Phase3: SKIPPED | SFT: Oct 26 13:09 (142.2h) | DPO (Final): Oct 26 13:43 (142.7h) [2026-10-20 15:00:22][INFO][training.pretrain_engine] Starting global step 510 (accumulating 32 micro-batches)... [2026-10-20 15:00:22][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 15:00:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5643 [2026-10-20 15:00:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3471 [2026-10-20 15:00:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3017 [2026-10-20 15:00:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9910 [2026-10-20 15:00:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8232 [2026-10-20 15:00:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3319 [2026-10-20 15:00:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8118 [2026-10-20 15:00:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7370 [2026-10-20 15:01:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8226 [2026-10-20 15:01:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0337 [2026-10-20 15:01:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8153 [2026-10-20 15:01:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0745 [2026-10-20 15:01:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1233 [2026-10-20 15:01:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3172 [2026-10-20 15:01:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1231 [2026-10-20 15:01:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2687 [2026-10-20 15:01:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6021 [2026-10-20 15:01:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6561 [2026-10-20 15:01:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1625 [2026-10-20 15:01:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9696 [2026-10-20 15:02:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7604 [2026-10-20 15:02:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6370 [2026-10-20 15:02:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7893 [2026-10-20 15:02:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1304 [2026-10-20 15:02:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9006 [2026-10-20 15:02:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8433 [2026-10-20 15:02:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5520 [2026-10-20 15:02:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0293 [2026-10-20 15:02:41][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,582 | MFU: 1.8% [2026-10-20 15:02:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5718 [2026-10-20 15:02:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5698 [2026-10-20 15:02:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9293 [2026-10-20 15:02:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9980 [2026-10-20 15:02:45][INFO][training.pretrain_engine] [Ph2|Step 510] loss=5.8184 | ppl=336.44 | aux=0.02509 | lr=3.00e-04 | gnorm=1.469 | tok/s=29,233 | MFU=7.0% | step=143481ms | tokens=0.542B | mem=4.2/32.7GB | cost=$268.01/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 08:25 (137.4h) | Phase3: SKIPPED | SFT: Oct 26 13:10 (142.1h) | DPO (Final): Oct 26 13:45 (142.7h) [2026-10-20 15:02:45][INFO][training.pretrain_engine] Starting global step 511 (accumulating 32 micro-batches)... [2026-10-20 15:02:45][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 15:03:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9547 [2026-10-20 15:03:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2334 [2026-10-20 15:03:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1800 [2026-10-20 15:03:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3040 [2026-10-20 15:03:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7041 [2026-10-20 15:03:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7986 [2026-10-20 15:03:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1469 [2026-10-20 15:03:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2325 [2026-10-20 15:03:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4580 [2026-10-20 15:03:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3102 [2026-10-20 15:03:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2331 [2026-10-20 15:03:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6369 [2026-10-20 15:03:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2312 [2026-10-20 15:03:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9178 [2026-10-20 15:03:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1030 [2026-10-20 15:03:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0523 [2026-10-20 15:04:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.6054 [2026-10-20 15:04:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9169 [2026-10-20 15:04:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3588 [2026-10-20 15:04:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4706 [2026-10-20 15:04:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1837 [2026-10-20 15:04:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1234 [2026-10-20 15:04:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6169 [2026-10-20 15:04:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8795 [2026-10-20 15:04:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6857 [2026-10-20 15:04:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0167 [2026-10-20 15:04:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9172 [2026-10-20 15:04:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7750 [2026-10-20 15:05:04][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,577 | MFU: 1.8% [2026-10-20 15:05:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9491 [2026-10-20 15:05:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9977 [2026-10-20 15:05:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9363 [2026-10-20 15:05:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1312 [2026-10-20 15:05:09][INFO][training.pretrain_engine] [Ph2|Step 511] loss=5.9724 | ppl=392.44 | aux=0.02509 | lr=3.00e-04 | gnorm=0.852 | tok/s=29,233 | MFU=7.0% | step=143479ms | tokens=0.543B | mem=4.2/32.7GB | cost=$268.53/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 08:27 (137.4h) | Phase3: SKIPPED | SFT: Oct 26 13:12 (142.1h) | DPO (Final): Oct 26 13:46 (142.7h) [2026-10-20 15:05:09][INFO][training.pretrain_engine] Starting global step 512 (accumulating 32 micro-batches)... [2026-10-20 15:05:09][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 15:05:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5691 [2026-10-20 15:05:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8141 [2026-10-20 15:05:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3119 [2026-10-20 15:05:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6498 [2026-10-20 15:05:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2156 [2026-10-20 15:05:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8809 [2026-10-20 15:05:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8178 [2026-10-20 15:05:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9491 [2026-10-20 15:06:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6899 [2026-10-20 15:06:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1749 [2026-10-20 15:06:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0117 [2026-10-20 15:06:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8193 [2026-10-20 15:06:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9431 [2026-10-20 15:06:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9460 [2026-10-20 15:06:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7512 [2026-10-20 15:06:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9067 [2026-10-20 15:06:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8674 [2026-10-20 15:06:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0404 [2026-10-20 15:06:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7900 [2026-10-20 15:06:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7761 [2026-10-20 15:06:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5742 [2026-10-20 15:06:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8466 [2026-10-20 15:06:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7755 [2026-10-20 15:06:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0051 [2026-10-20 15:07:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4765 [2026-10-20 15:07:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8857 [2026-10-20 15:07:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1714 [2026-10-20 15:07:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9543 [2026-10-20 15:07:27][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,556 | MFU: 1.8% [2026-10-20 15:07:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0262 [2026-10-20 15:07:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0015 [2026-10-20 15:07:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7787 [2026-10-20 15:07:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3870 [2026-10-20 15:07:32][INFO][training.pretrain_engine] [Ph2|Step 512] loss=5.8500 | ppl=347.25 | aux=0.02511 | lr=3.00e-04 | gnorm=1.219 | tok/s=29,234 | MFU=7.0% | step=143473ms | tokens=0.544B | mem=4.2/32.7GB | cost=$269.06/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 08:28 (137.4h) | Phase3: SKIPPED | SFT: Oct 26 13:14 (142.1h) | DPO (Final): Oct 26 13:48 (142.7h) [2026-10-20 15:07:32][INFO][training.pretrain_engine] Starting global step 513 (accumulating 32 micro-batches)... [2026-10-20 15:07:32][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 15:07:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2169 [2026-10-20 15:07:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6376 [2026-10-20 15:07:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3701 [2026-10-20 15:07:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4964 [2026-10-20 15:08:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8093 [2026-10-20 15:08:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2059 [2026-10-20 15:08:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1754 [2026-10-20 15:08:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5449 [2026-10-20 15:08:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0796 [2026-10-20 15:08:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7835 [2026-10-20 15:08:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6339 [2026-10-20 15:08:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0195 [2026-10-20 15:08:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7837 [2026-10-20 15:08:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6864 [2026-10-20 15:08:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8182 [2026-10-20 15:08:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7244 [2026-10-20 15:09:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2324 [2026-10-20 15:09:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2407 [2026-10-20 15:09:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9310 [2026-10-20 15:09:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7392 [2026-10-20 15:09:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9887 [2026-10-20 15:09:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2186 [2026-10-20 15:09:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0870 [2026-10-20 15:09:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1348 [2026-10-20 15:09:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8270 [2026-10-20 15:09:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2248 [2026-10-20 15:09:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1214 [2026-10-20 15:09:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4158 [2026-10-20 15:09:50][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,569 | MFU: 1.8% [2026-10-20 15:09:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6376 [2026-10-20 15:09:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4865 [2026-10-20 15:09:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7345 [2026-10-20 15:09:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4740 [2026-10-20 15:09:55][INFO][training.pretrain_engine] [Ph2|Step 513] loss=5.8459 | ppl=345.81 | aux=0.02510 | lr=3.00e-04 | gnorm=0.855 | tok/s=29,236 | MFU=7.0% | step=143462ms | tokens=0.545B | mem=4.2/32.7GB | cost=$269.58/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 08:30 (137.3h) | Phase3: SKIPPED | SFT: Oct 26 13:15 (142.1h) | DPO (Final): Oct 26 13:49 (142.7h) [2026-10-20 15:09:55][INFO][training.pretrain_engine] Starting global step 514 (accumulating 32 micro-batches)... [2026-10-20 15:09:55][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 15:10:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9638 [2026-10-20 15:10:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4825 [2026-10-20 15:10:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3106 [2026-10-20 15:10:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5887 [2026-10-20 15:10:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9318 [2026-10-20 15:10:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8208 [2026-10-20 15:10:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8909 [2026-10-20 15:10:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7892 [2026-10-20 15:10:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0171 [2026-10-20 15:10:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7020 [2026-10-20 15:10:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7898 [2026-10-20 15:10:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8703 [2026-10-20 15:11:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3109 [2026-10-20 15:11:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5474 [2026-10-20 15:11:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6460 [2026-10-20 15:11:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8745 [2026-10-20 15:11:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1969 [2026-10-20 15:11:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2110 [2026-10-20 15:11:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8484 [2026-10-20 15:11:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9648 [2026-10-20 15:11:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9192 [2026-10-20 15:11:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2629 [2026-10-20 15:11:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0919 [2026-10-20 15:11:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4783 [2026-10-20 15:12:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7955 [2026-10-20 15:12:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2828 [2026-10-20 15:12:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8211 [2026-10-20 15:12:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4664 [2026-10-20 15:12:14][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,575 | MFU: 1.8% [2026-10-20 15:12:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7649 [2026-10-20 15:12:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1644 [2026-10-20 15:12:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9289 [2026-10-20 15:12:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8348 [2026-10-20 15:12:18][INFO][training.pretrain_engine] [Ph2|Step 514] loss=5.8161 | ppl=335.65 | aux=0.02510 | lr=3.00e-04 | gnorm=0.930 | tok/s=29,237 | MFU=7.0% | step=143458ms | tokens=0.546B | mem=4.2/32.7GB | cost=$270.10/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 08:31 (137.3h) | Phase3: SKIPPED | SFT: Oct 26 13:16 (142.1h) | DPO (Final): Oct 26 13:50 (142.6h) [2026-10-20 15:12:18][INFO][training.pretrain_engine] Starting global step 515 (accumulating 32 micro-batches)... [2026-10-20 15:12:18][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 15:12:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7907 [2026-10-20 15:12:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5329 [2026-10-20 15:12:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7394 [2026-10-20 15:12:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9440 [2026-10-20 15:12:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1175 [2026-10-20 15:12:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0310 [2026-10-20 15:12:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7581 [2026-10-20 15:12:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0845 [2026-10-20 15:13:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5300 [2026-10-20 15:13:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2589 [2026-10-20 15:13:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0981 [2026-10-20 15:13:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4434 [2026-10-20 15:13:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9684 [2026-10-20 15:13:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9089 [2026-10-20 15:13:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0357 [2026-10-20 15:13:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2005 [2026-10-20 15:13:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0885 [2026-10-20 15:13:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8099 [2026-10-20 15:13:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5895 [2026-10-20 15:13:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8997 [2026-10-20 15:14:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5071 [2026-10-20 15:14:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3029 [2026-10-20 15:14:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5984 [2026-10-20 15:14:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1540 [2026-10-20 15:14:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4842 [2026-10-20 15:14:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7053 [2026-10-20 15:14:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6798 [2026-10-20 15:14:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2654 [2026-10-20 15:14:37][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,567 | MFU: 1.8% [2026-10-20 15:14:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1103 [2026-10-20 15:14:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1225 [2026-10-20 15:14:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8653 [2026-10-20 15:14:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4856 [2026-10-20 15:14:42][INFO][training.pretrain_engine] [Ph2|Step 515] loss=5.8447 | ppl=345.39 | aux=0.02509 | lr=3.00e-04 | gnorm=0.848 | tok/s=29,238 | MFU=7.0% | step=143452ms | tokens=0.547B | mem=4.2/32.7GB | cost=$270.63/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 08:33 (137.3h) | Phase3: SKIPPED | SFT: Oct 26 13:18 (142.1h) | DPO (Final): Oct 26 13:52 (142.6h) [2026-10-20 15:14:45][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step515 | Tokens: 0.547B | Spent: $270.64 | Remaining: $129.36 [2026-10-20 15:14:45][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step515 [2026-10-20 15:14:45][INFO][training.pretrain_engine] Starting global step 516 (accumulating 32 micro-batches)... [2026-10-20 15:14:45][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 15:14:45][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 15:14:45][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 15:14:45][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 15:14:45][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step515 to Phase-Technologies/forge-500m-pretrain (path: phase2_step515)... [2026-10-20 15:14:45][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 15:14:54][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 15:14:54][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step515 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 15:14:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8654 [2026-10-20 15:14:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9586 [2026-10-20 15:15:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7796 [2026-10-20 15:15:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2655 [2026-10-20 15:15:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6566 [2026-10-20 15:15:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7078 [2026-10-20 15:15:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9586 [2026-10-20 15:15:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8130 [2026-10-20 15:15:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4619 [2026-10-20 15:15:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1297 [2026-10-20 15:15:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2753 [2026-10-20 15:15:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3629 [2026-10-20 15:15:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6687 [2026-10-20 15:15:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0733 [2026-10-20 15:15:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7474 [2026-10-20 15:15:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9747 [2026-10-20 15:16:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8727 [2026-10-20 15:16:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.9353 [2026-10-20 15:16:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4520 [2026-10-20 15:16:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1482 [2026-10-20 15:16:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1093 [2026-10-20 15:16:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4699 [2026-10-20 15:16:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1797 [2026-10-20 15:16:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0402 [2026-10-20 15:16:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4572 [2026-10-20 15:16:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4315 [2026-10-20 15:16:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2545 [2026-10-20 15:16:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1982 [2026-10-20 15:17:04][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,541 | MFU: 1.8% [2026-10-20 15:17:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1984 [2026-10-20 15:17:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7610 [2026-10-20 15:17:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7120 [2026-10-20 15:17:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7014 [2026-10-20 15:17:08][INFO][training.pretrain_engine] [Ph2|Step 516] loss=5.8349 | ppl=342.04 | aux=0.02511 | lr=3.00e-04 | gnorm=0.762 | tok/s=29,239 | MFU=7.0% | step=143448ms | tokens=0.548B | mem=4.2/32.7GB | cost=$271.16/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 08:34 (137.3h) | Phase3: SKIPPED | SFT: Oct 26 13:19 (142.0h) | DPO (Final): Oct 26 13:53 (142.6h) [2026-10-20 15:17:08][INFO][training.pretrain_engine] Starting global step 517 (accumulating 32 micro-batches)... [2026-10-20 15:17:08][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 15:17:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3975 [2026-10-20 15:17:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6830 [2026-10-20 15:17:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0568 [2026-10-20 15:17:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5352 [2026-10-20 15:17:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8234 [2026-10-20 15:17:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9611 [2026-10-20 15:17:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2932 [2026-10-20 15:17:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8085 [2026-10-20 15:18:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8530 [2026-10-20 15:18:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9224 [2026-10-20 15:18:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5186 [2026-10-20 15:18:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2202 [2026-10-20 15:18:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0193 [2026-10-20 15:18:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9722 [2026-10-20 15:18:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0553 [2026-10-20 15:18:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3097 [2026-10-20 15:18:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1440 [2026-10-20 15:18:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8428 [2026-10-20 15:18:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1471 [2026-10-20 15:18:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0853 [2026-10-20 15:18:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0738 [2026-10-20 15:18:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6956 [2026-10-20 15:18:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7940 [2026-10-20 15:18:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7343 [2026-10-20 15:19:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3677 [2026-10-20 15:19:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9328 [2026-10-20 15:19:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9813 [2026-10-20 15:19:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9943 [2026-10-20 15:19:27][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,568 | MFU: 1.8% [2026-10-20 15:19:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4524 [2026-10-20 15:19:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3385 [2026-10-20 15:19:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6981 [2026-10-20 15:19:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2703 [2026-10-20 15:19:31][INFO][training.pretrain_engine] [Ph2|Step 517] loss=5.8561 | ppl=349.36 | aux=0.02508 | lr=3.00e-04 | gnorm=0.695 | tok/s=29,240 | MFU=7.0% | step=143444ms | tokens=0.549B | mem=4.2/32.7GB | cost=$271.69/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 08:36 (137.3h) | Phase3: SKIPPED | SFT: Oct 26 13:21 (142.0h) | DPO (Final): Oct 26 13:55 (142.6h) [2026-10-20 15:19:31][INFO][training.pretrain_engine] Starting global step 518 (accumulating 32 micro-batches)... [2026-10-20 15:19:31][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 15:19:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0658 [2026-10-20 15:19:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1917 [2026-10-20 15:19:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0044 [2026-10-20 15:19:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8181 [2026-10-20 15:20:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6498 [2026-10-20 15:20:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7356 [2026-10-20 15:20:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8995 [2026-10-20 15:20:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0450 [2026-10-20 15:20:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9554 [2026-10-20 15:20:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7399 [2026-10-20 15:20:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7663 [2026-10-20 15:20:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0901 [2026-10-20 15:20:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0540 [2026-10-20 15:20:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9923 [2026-10-20 15:20:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3841 [2026-10-20 15:20:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4116 [2026-10-20 15:21:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4184 [2026-10-20 15:21:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0035 [2026-10-20 15:21:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1105 [2026-10-20 15:21:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=7.0272 [2026-10-20 15:21:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7265 [2026-10-20 15:21:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8795 [2026-10-20 15:21:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0100 [2026-10-20 15:21:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7798 [2026-10-20 15:21:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4295 [2026-10-20 15:21:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3388 [2026-10-20 15:21:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6009 [2026-10-20 15:21:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6920 [2026-10-20 15:21:50][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,558 | MFU: 1.8% [2026-10-20 15:21:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1896 [2026-10-20 15:21:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9371 [2026-10-20 15:21:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8579 [2026-10-20 15:21:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9848 [2026-10-20 15:21:55][INFO][training.pretrain_engine] [Ph2|Step 518] loss=5.8618 | ppl=351.35 | aux=0.02509 | lr=3.00e-04 | gnorm=0.867 | tok/s=29,242 | MFU=7.0% | step=143436ms | tokens=0.551B | mem=4.2/32.7GB | cost=$272.21/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 08:37 (137.3h) | Phase3: SKIPPED | SFT: Oct 26 13:22 (142.0h) | DPO (Final): Oct 26 13:56 (142.6h) [2026-10-20 15:21:55][INFO][training.pretrain_engine] Starting global step 519 (accumulating 32 micro-batches)... [2026-10-20 15:21:55][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 15:22:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1463 [2026-10-20 15:22:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8794 [2026-10-20 15:22:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3984 [2026-10-20 15:22:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4836 [2026-10-20 15:22:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8252 [2026-10-20 15:22:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4613 [2026-10-20 15:22:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1451 [2026-10-20 15:22:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9046 [2026-10-20 15:22:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7745 [2026-10-20 15:22:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8490 [2026-10-20 15:22:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1144 [2026-10-20 15:22:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8802 [2026-10-20 15:23:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8650 [2026-10-20 15:23:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2835 [2026-10-20 15:23:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6783 [2026-10-20 15:23:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3150 [2026-10-20 15:23:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7628 [2026-10-20 15:23:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6427 [2026-10-20 15:23:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7938 [2026-10-20 15:23:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7075 [2026-10-20 15:23:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6871 [2026-10-20 15:23:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8030 [2026-10-20 15:23:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6854 [2026-10-20 15:23:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1003 [2026-10-20 15:24:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0718 [2026-10-20 15:24:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1915 [2026-10-20 15:24:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2504 [2026-10-20 15:24:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6183 [2026-10-20 15:24:14][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,549 | MFU: 1.8% [2026-10-20 15:24:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5712 [2026-10-20 15:24:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3908 [2026-10-20 15:24:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9711 [2026-10-20 15:24:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5912 [2026-10-20 15:24:18][INFO][training.pretrain_engine] [Ph2|Step 519] loss=5.9124 | ppl=369.61 | aux=0.02510 | lr=3.00e-04 | gnorm=0.871 | tok/s=29,242 | MFU=7.0% | step=143435ms | tokens=0.552B | mem=4.2/32.7GB | cost=$272.74/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 08:39 (137.3h) | Phase3: SKIPPED | SFT: Oct 26 13:24 (142.0h) | DPO (Final): Oct 26 13:58 (142.6h) [2026-10-20 15:24:18][INFO][training.pretrain_engine] Starting global step 520 (accumulating 32 micro-batches)... [2026-10-20 15:24:18][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 15:24:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6200 [2026-10-20 15:24:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9760 [2026-10-20 15:24:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9555 [2026-10-20 15:24:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0704 [2026-10-20 15:24:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2957 [2026-10-20 15:24:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9025 [2026-10-20 15:24:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4913 [2026-10-20 15:24:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9415 [2026-10-20 15:25:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7462 [2026-10-20 15:25:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1634 [2026-10-20 15:25:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5920 [2026-10-20 15:25:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9428 [2026-10-20 15:25:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7300 [2026-10-20 15:25:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9346 [2026-10-20 15:25:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0484 [2026-10-20 15:25:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7335 [2026-10-20 15:25:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0233 [2026-10-20 15:25:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1147 [2026-10-20 15:25:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0143 [2026-10-20 15:25:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7231 [2026-10-20 15:26:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8589 [2026-10-20 15:26:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4112 [2026-10-20 15:26:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=7.1768 [2026-10-20 15:26:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8351 [2026-10-20 15:26:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7914 [2026-10-20 15:26:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9734 [2026-10-20 15:26:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8238 [2026-10-20 15:26:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6801 [2026-10-20 15:26:37][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,569 | MFU: 1.8% [2026-10-20 15:26:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1336 [2026-10-20 15:26:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8974 [2026-10-20 15:26:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5713 [2026-10-20 15:26:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7668 [2026-10-20 15:26:41][INFO][training.pretrain_engine] [Ph2|Step 520] loss=5.6674 | ppl=289.28 | aux=0.02509 | lr=3.00e-04 | gnorm=0.996 | tok/s=29,242 | MFU=7.0% | step=143432ms | tokens=0.553B | mem=4.2/32.7GB | cost=$273.26/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 08:40 (137.2h) | Phase3: SKIPPED | SFT: Oct 26 13:25 (142.0h) | DPO (Final): Oct 26 14:00 (142.6h) [2026-10-20 15:26:41][INFO][training.pretrain_engine] Starting global step 521 (accumulating 32 micro-batches)... [2026-10-20 15:26:41][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 15:26:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0189 [2026-10-20 15:26:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5712 [2026-10-20 15:26:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8909 [2026-10-20 15:26:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5325 [2026-10-20 15:27:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5279 [2026-10-20 15:27:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3773 [2026-10-20 15:27:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7963 [2026-10-20 15:27:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8504 [2026-10-20 15:27:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6047 [2026-10-20 15:27:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5315 [2026-10-20 15:27:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6715 [2026-10-20 15:27:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6074 [2026-10-20 15:27:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4780 [2026-10-20 15:27:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0598 [2026-10-20 15:27:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8524 [2026-10-20 15:27:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2781 [2026-10-20 15:28:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0384 [2026-10-20 15:28:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8376 [2026-10-20 15:28:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0558 [2026-10-20 15:28:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5949 [2026-10-20 15:28:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6091 [2026-10-20 15:28:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4112 [2026-10-20 15:28:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4205 [2026-10-20 15:28:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8651 [2026-10-20 15:28:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9557 [2026-10-20 15:28:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8724 [2026-10-20 15:28:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9895 [2026-10-20 15:28:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0013 [2026-10-20 15:29:00][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,561 | MFU: 1.8% [2026-10-20 15:29:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2335 [2026-10-20 15:29:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0691 [2026-10-20 15:29:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0912 [2026-10-20 15:29:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7973 [2026-10-20 15:29:05][INFO][training.pretrain_engine] [Ph2|Step 521] loss=5.8210 | ppl=337.31 | aux=0.02510 | lr=3.00e-04 | gnorm=0.910 | tok/s=29,243 | MFU=7.0% | step=143430ms | tokens=0.554B | mem=4.2/32.7GB | cost=$273.78/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 08:42 (137.2h) | Phase3: SKIPPED | SFT: Oct 26 13:27 (142.0h) | DPO (Final): Oct 26 14:01 (142.5h) [2026-10-20 15:29:05][INFO][training.pretrain_engine] Starting global step 522 (accumulating 32 micro-batches)... [2026-10-20 15:29:05][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 15:29:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8831 [2026-10-20 15:29:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4600 [2026-10-20 15:29:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5440 [2026-10-20 15:29:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6655 [2026-10-20 15:29:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7907 [2026-10-20 15:29:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7302 [2026-10-20 15:29:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5916 [2026-10-20 15:29:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8428 [2026-10-20 15:29:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0600 [2026-10-20 15:29:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0837 [2026-10-20 15:29:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7156 [2026-10-20 15:29:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9796 [2026-10-20 15:30:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0463 [2026-10-20 15:30:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8831 [2026-10-20 15:30:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0079 [2026-10-20 15:30:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7534 [2026-10-20 15:30:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2005 [2026-10-20 15:30:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9872 [2026-10-20 15:30:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2431 [2026-10-20 15:30:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2325 [2026-10-20 15:30:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5372 [2026-10-20 15:30:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3835 [2026-10-20 15:30:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7759 [2026-10-20 15:30:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0909 [2026-10-20 15:31:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.9411 [2026-10-20 15:31:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5541 [2026-10-20 15:31:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1211 [2026-10-20 15:31:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4362 [2026-10-20 15:31:23][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,569 | MFU: 1.8% [2026-10-20 15:31:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0615 [2026-10-20 15:31:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8256 [2026-10-20 15:31:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1613 [2026-10-20 15:31:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8939 [2026-10-20 15:31:28][INFO][training.pretrain_engine] [Ph2|Step 522] loss=5.9601 | ppl=387.66 | aux=0.02509 | lr=3.00e-04 | gnorm=0.781 | tok/s=29,244 | MFU=7.0% | step=143423ms | tokens=0.555B | mem=4.2/32.7GB | cost=$274.31/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 08:44 (137.2h) | Phase3: SKIPPED | SFT: Oct 26 13:28 (142.0h) | DPO (Final): Oct 26 14:03 (142.5h) [2026-10-20 15:31:28][INFO][training.pretrain_engine] Starting global step 523 (accumulating 32 micro-batches)... [2026-10-20 15:31:28][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 15:31:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8467 [2026-10-20 15:31:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1514 [2026-10-20 15:31:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8089 [2026-10-20 15:31:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6023 [2026-10-20 15:32:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6990 [2026-10-20 15:32:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6660 [2026-10-20 15:32:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7055 [2026-10-20 15:32:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8451 [2026-10-20 15:32:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9365 [2026-10-20 15:32:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4454 [2026-10-20 15:32:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9079 [2026-10-20 15:32:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9271 [2026-10-20 15:32:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0281 [2026-10-20 15:32:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2071 [2026-10-20 15:32:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0121 [2026-10-20 15:32:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2970 [2026-10-20 15:32:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7725 [2026-10-20 15:32:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3487 [2026-10-20 15:32:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3322 [2026-10-20 15:32:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6600 [2026-10-20 15:33:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2321 [2026-10-20 15:33:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8854 [2026-10-20 15:33:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5806 [2026-10-20 15:33:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1998 [2026-10-20 15:33:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1675 [2026-10-20 15:33:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8692 [2026-10-20 15:33:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4616 [2026-10-20 15:33:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1848 [2026-10-20 15:33:46][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,590 | MFU: 1.8% [2026-10-20 15:33:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1278 [2026-10-20 15:33:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7717 [2026-10-20 15:33:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7963 [2026-10-20 15:33:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6635 [2026-10-20 15:33:51][INFO][training.pretrain_engine] [Ph2|Step 523] loss=5.8082 | ppl=333.03 | aux=0.02511 | lr=3.00e-04 | gnorm=0.855 | tok/s=29,245 | MFU=7.0% | step=143421ms | tokens=0.556B | mem=4.2/32.7GB | cost=$274.83/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 08:45 (137.2h) | Phase3: SKIPPED | SFT: Oct 26 13:30 (141.9h) | DPO (Final): Oct 26 14:04 (142.5h) [2026-10-20 15:33:51][INFO][training.pretrain_engine] Starting global step 524 (accumulating 32 micro-batches)... [2026-10-20 15:33:51][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 15:34:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0335 [2026-10-20 15:34:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2207 [2026-10-20 15:34:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0230 [2026-10-20 15:34:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8137 [2026-10-20 15:34:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6959 [2026-10-20 15:34:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3851 [2026-10-20 15:34:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6758 [2026-10-20 15:34:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3035 [2026-10-20 15:34:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9849 [2026-10-20 15:34:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5074 [2026-10-20 15:34:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8447 [2026-10-20 15:34:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6443 [2026-10-20 15:35:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3007 [2026-10-20 15:35:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5767 [2026-10-20 15:35:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9044 [2026-10-20 15:35:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8849 [2026-10-20 15:35:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9316 [2026-10-20 15:35:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8479 [2026-10-20 15:35:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5518 [2026-10-20 15:35:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8098 [2026-10-20 15:35:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6956 [2026-10-20 15:35:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9688 [2026-10-20 15:35:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8968 [2026-10-20 15:35:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9394 [2026-10-20 15:35:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9168 [2026-10-20 15:35:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7920 [2026-10-20 15:35:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1528 [2026-10-20 15:35:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9761 [2026-10-20 15:36:10][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,565 | MFU: 1.8% [2026-10-20 15:36:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7424 [2026-10-20 15:36:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7581 [2026-10-20 15:36:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4877 [2026-10-20 15:36:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9910 [2026-10-20 15:36:15][INFO][training.pretrain_engine] [Ph2|Step 524] loss=5.8632 | ppl=351.83 | aux=0.02509 | lr=3.00e-04 | gnorm=0.816 | tok/s=29,244 | MFU=7.0% | step=143426ms | tokens=0.557B | mem=4.2/32.7GB | cost=$275.35/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 08:47 (137.2h) | Phase3: SKIPPED | SFT: Oct 26 13:32 (141.9h) | DPO (Final): Oct 26 14:06 (142.5h) [2026-10-20 15:36:18][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step524 | Tokens: 0.557B | Spent: $275.37 | Remaining: $124.63 [2026-10-20 15:36:18][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step524 [2026-10-20 15:36:18][INFO][training.pretrain_engine] Starting global step 525 (accumulating 32 micro-batches)... [2026-10-20 15:36:18][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 15:36:18][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 15:36:18][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 15:36:18][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 15:36:18][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step524 to Phase-Technologies/forge-500m-pretrain (path: phase2_step524)... [2026-10-20 15:36:18][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 15:36:28][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 15:36:28][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step524 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 15:36:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8192 [2026-10-20 15:36:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1735 [2026-10-20 15:36:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7377 [2026-10-20 15:36:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1230 [2026-10-20 15:36:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2316 [2026-10-20 15:36:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2771 [2026-10-20 15:36:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1739 [2026-10-20 15:36:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1961 [2026-10-20 15:37:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9083 [2026-10-20 15:37:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3995 [2026-10-20 15:37:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6650 [2026-10-20 15:37:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9156 [2026-10-20 15:37:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8276 [2026-10-20 15:37:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1503 [2026-10-20 15:37:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9580 [2026-10-20 15:37:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5391 [2026-10-20 15:37:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0850 [2026-10-20 15:37:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1004 [2026-10-20 15:37:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3254 [2026-10-20 15:37:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2207 [2026-10-20 15:38:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4839 [2026-10-20 15:38:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0654 [2026-10-20 15:38:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5738 [2026-10-20 15:38:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4097 [2026-10-20 15:38:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1254 [2026-10-20 15:38:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9860 [2026-10-20 15:38:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8222 [2026-10-20 15:38:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2066 [2026-10-20 15:38:37][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,560 | MFU: 1.8% [2026-10-20 15:38:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1599 [2026-10-20 15:38:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4551 [2026-10-20 15:38:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4355 [2026-10-20 15:38:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3017 [2026-10-20 15:38:41][INFO][training.pretrain_engine] [Ph2|Step 525] loss=5.9253 | ppl=374.38 | aux=0.02508 | lr=3.00e-04 | gnorm=0.777 | tok/s=29,245 | MFU=7.0% | step=143422ms | tokens=0.558B | mem=4.2/32.7GB | cost=$275.89/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 08:49 (137.2h) | Phase3: SKIPPED | SFT: Oct 26 13:34 (141.9h) | DPO (Final): Oct 26 14:08 (142.5h) [2026-10-20 15:38:41][INFO][training.pretrain_engine] Starting global step 526 (accumulating 32 micro-batches)... [2026-10-20 15:38:41][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 15:38:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9465 [2026-10-20 15:38:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0973 [2026-10-20 15:38:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2551 [2026-10-20 15:38:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8153 [2026-10-20 15:39:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8944 [2026-10-20 15:39:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9436 [2026-10-20 15:39:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1970 [2026-10-20 15:39:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9136 [2026-10-20 15:39:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4859 [2026-10-20 15:39:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3454 [2026-10-20 15:39:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4124 [2026-10-20 15:39:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4000 [2026-10-20 15:39:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6838 [2026-10-20 15:39:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0454 [2026-10-20 15:39:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8024 [2026-10-20 15:39:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2222 [2026-10-20 15:40:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7015 [2026-10-20 15:40:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9577 [2026-10-20 15:40:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1823 [2026-10-20 15:40:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9478 [2026-10-20 15:40:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9249 [2026-10-20 15:40:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6340 [2026-10-20 15:40:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0032 [2026-10-20 15:40:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7669 [2026-10-20 15:40:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6631 [2026-10-20 15:40:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6241 [2026-10-20 15:40:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9264 [2026-10-20 15:40:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9201 [2026-10-20 15:41:00][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,557 | MFU: 1.8% [2026-10-20 15:41:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6617 [2026-10-20 15:41:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5135 [2026-10-20 15:41:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8953 [2026-10-20 15:41:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3377 [2026-10-20 15:41:05][INFO][training.pretrain_engine] [Ph2|Step 526] loss=5.8642 | ppl=352.21 | aux=0.02509 | lr=3.00e-04 | gnorm=0.871 | tok/s=29,245 | MFU=7.0% | step=143419ms | tokens=0.559B | mem=4.2/32.7GB | cost=$276.42/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 08:51 (137.2h) | Phase3: SKIPPED | SFT: Oct 26 13:35 (141.9h) | DPO (Final): Oct 26 14:10 (142.5h) [2026-10-20 15:41:05][INFO][training.pretrain_engine] Starting global step 527 (accumulating 32 micro-batches)... [2026-10-20 15:41:05][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 15:41:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8185 [2026-10-20 15:41:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9503 [2026-10-20 15:41:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4419 [2026-10-20 15:41:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6217 [2026-10-20 15:41:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0714 [2026-10-20 15:41:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9131 [2026-10-20 15:41:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9023 [2026-10-20 15:41:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5903 [2026-10-20 15:41:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8832 [2026-10-20 15:41:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6279 [2026-10-20 15:41:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7896 [2026-10-20 15:41:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8260 [2026-10-20 15:42:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2570 [2026-10-20 15:42:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7557 [2026-10-20 15:42:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3154 [2026-10-20 15:42:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7226 [2026-10-20 15:42:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8535 [2026-10-20 15:42:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1969 [2026-10-20 15:42:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7612 [2026-10-20 15:42:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8578 [2026-10-20 15:42:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0037 [2026-10-20 15:42:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8129 [2026-10-20 15:42:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4361 [2026-10-20 15:42:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7444 [2026-10-20 15:43:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4446 [2026-10-20 15:43:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9936 [2026-10-20 15:43:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0697 [2026-10-20 15:43:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7556 [2026-10-20 15:43:23][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,576 | MFU: 1.8% [2026-10-20 15:43:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7873 [2026-10-20 15:43:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1829 [2026-10-20 15:43:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1086 [2026-10-20 15:43:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2989 [2026-10-20 15:43:28][INFO][training.pretrain_engine] [Ph2|Step 527] loss=5.9066 | ppl=367.45 | aux=0.02510 | lr=3.00e-04 | gnorm=0.754 | tok/s=29,247 | MFU=7.0% | step=143411ms | tokens=0.560B | mem=4.2/32.7GB | cost=$276.94/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 08:52 (137.1h) | Phase3: SKIPPED | SFT: Oct 26 13:37 (141.9h) | DPO (Final): Oct 26 14:11 (142.5h) [2026-10-20 15:43:28][INFO][training.pretrain_engine] Starting global step 528 (accumulating 32 micro-batches)... [2026-10-20 15:43:28][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 15:43:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3331 [2026-10-20 15:43:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8021 [2026-10-20 15:43:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3317 [2026-10-20 15:43:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6582 [2026-10-20 15:44:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8869 [2026-10-20 15:44:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0313 [2026-10-20 15:44:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6576 [2026-10-20 15:44:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0079 [2026-10-20 15:44:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7327 [2026-10-20 15:44:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7905 [2026-10-20 15:44:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2710 [2026-10-20 15:44:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8611 [2026-10-20 15:44:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6268 [2026-10-20 15:44:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2221 [2026-10-20 15:44:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7198 [2026-10-20 15:44:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4730 [2026-10-20 15:44:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8000 [2026-10-20 15:44:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6272 [2026-10-20 15:44:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4288 [2026-10-20 15:44:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2629 [2026-10-20 15:45:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1134 [2026-10-20 15:45:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5702 [2026-10-20 15:45:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9350 [2026-10-20 15:45:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5437 [2026-10-20 15:45:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5716 [2026-10-20 15:45:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6985 [2026-10-20 15:45:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9539 [2026-10-20 15:45:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9470 [2026-10-20 15:45:46][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,579 | MFU: 1.8% [2026-10-20 15:45:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8094 [2026-10-20 15:45:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5102 [2026-10-20 15:45:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8213 [2026-10-20 15:45:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8464 [2026-10-20 15:45:51][INFO][training.pretrain_engine] [Ph2|Step 528] loss=5.8135 | ppl=334.78 | aux=0.02509 | lr=3.00e-04 | gnorm=0.984 | tok/s=29,247 | MFU=7.0% | step=143409ms | tokens=0.561B | mem=4.2/32.7GB | cost=$277.46/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 08:54 (137.1h) | Phase3: SKIPPED | SFT: Oct 26 13:38 (141.9h) | DPO (Final): Oct 26 14:13 (142.5h) [2026-10-20 15:45:51][INFO][training.pretrain_engine] Starting global step 529 (accumulating 32 micro-batches)... [2026-10-20 15:45:51][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 15:46:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9240 [2026-10-20 15:46:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8306 [2026-10-20 15:46:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7512 [2026-10-20 15:46:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4319 [2026-10-20 15:46:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0065 [2026-10-20 15:46:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1404 [2026-10-20 15:46:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1195 [2026-10-20 15:46:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1482 [2026-10-20 15:46:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4609 [2026-10-20 15:46:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0380 [2026-10-20 15:46:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8974 [2026-10-20 15:46:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6861 [2026-10-20 15:47:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9247 [2026-10-20 15:47:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8750 [2026-10-20 15:47:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7391 [2026-10-20 15:47:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8782 [2026-10-20 15:47:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1435 [2026-10-20 15:47:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8080 [2026-10-20 15:47:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6325 [2026-10-20 15:47:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9317 [2026-10-20 15:47:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9741 [2026-10-20 15:47:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1344 [2026-10-20 15:47:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6643 [2026-10-20 15:47:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7881 [2026-10-20 15:47:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5418 [2026-10-20 15:47:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2430 [2026-10-20 15:47:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1618 [2026-10-20 15:47:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8922 [2026-10-20 15:48:10][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,544 | MFU: 1.8% [2026-10-20 15:48:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1068 [2026-10-20 15:48:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8015 [2026-10-20 15:48:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9211 [2026-10-20 15:48:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7149 [2026-10-20 15:48:15][INFO][training.pretrain_engine] [Ph2|Step 529] loss=5.8479 | ppl=346.52 | aux=0.02509 | lr=3.00e-04 | gnorm=1.227 | tok/s=29,247 | MFU=7.0% | step=143408ms | tokens=0.562B | mem=4.2/32.7GB | cost=$277.99/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 08:55 (137.1h) | Phase3: SKIPPED | SFT: Oct 26 13:40 (141.9h) | DPO (Final): Oct 26 14:14 (142.4h) [2026-10-20 15:48:15][INFO][training.pretrain_engine] Starting global step 530 (accumulating 32 micro-batches)... [2026-10-20 15:48:15][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 15:48:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7734 [2026-10-20 15:48:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0385 [2026-10-20 15:48:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3862 [2026-10-20 15:48:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6740 [2026-10-20 15:48:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9291 [2026-10-20 15:48:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9794 [2026-10-20 15:48:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7500 [2026-10-20 15:48:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0759 [2026-10-20 15:49:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7784 [2026-10-20 15:49:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7592 [2026-10-20 15:49:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2688 [2026-10-20 15:49:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7086 [2026-10-20 15:49:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6433 [2026-10-20 15:49:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9695 [2026-10-20 15:49:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5902 [2026-10-20 15:49:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0587 [2026-10-20 15:49:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4205 [2026-10-20 15:49:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3035 [2026-10-20 15:49:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1365 [2026-10-20 15:49:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0560 [2026-10-20 15:50:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9161 [2026-10-20 15:50:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2395 [2026-10-20 15:50:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9823 [2026-10-20 15:50:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3096 [2026-10-20 15:50:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3432 [2026-10-20 15:50:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4558 [2026-10-20 15:50:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6069 [2026-10-20 15:50:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.0942 [2026-10-20 15:50:33][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,574 | MFU: 1.8% [2026-10-20 15:50:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4601 [2026-10-20 15:50:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8537 [2026-10-20 15:50:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7155 [2026-10-20 15:50:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2632 [2026-10-20 15:50:38][INFO][training.pretrain_engine] [Ph2|Step 530] loss=5.7852 | ppl=325.44 | aux=0.02510 | lr=3.00e-04 | gnorm=0.836 | tok/s=29,248 | MFU=7.0% | step=143406ms | tokens=0.563B | mem=4.2/32.7GB | cost=$278.51/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 08:57 (137.1h) | Phase3: SKIPPED | SFT: Oct 26 13:42 (141.9h) | DPO (Final): Oct 26 14:16 (142.4h) [2026-10-20 15:50:38][INFO][training.pretrain_engine] Starting global step 531 (accumulating 32 micro-batches)... [2026-10-20 15:50:38][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 15:50:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0959 [2026-10-20 15:50:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8090 [2026-10-20 15:50:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9113 [2026-10-20 15:50:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7851 [2026-10-20 15:51:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8431 [2026-10-20 15:51:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4083 [2026-10-20 15:51:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1502 [2026-10-20 15:51:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3599 [2026-10-20 15:51:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0693 [2026-10-20 15:51:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1314 [2026-10-20 15:51:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1493 [2026-10-20 15:51:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5421 [2026-10-20 15:51:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6808 [2026-10-20 15:51:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6228 [2026-10-20 15:51:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7131 [2026-10-20 15:51:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5714 [2026-10-20 15:52:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7424 [2026-10-20 15:52:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0286 [2026-10-20 15:52:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9385 [2026-10-20 15:52:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9605 [2026-10-20 15:52:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8111 [2026-10-20 15:52:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8912 [2026-10-20 15:52:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4097 [2026-10-20 15:52:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7341 [2026-10-20 15:52:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0021 [2026-10-20 15:52:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4104 [2026-10-20 15:52:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8780 [2026-10-20 15:52:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9223 [2026-10-20 15:52:57][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,538 | MFU: 1.8% [2026-10-20 15:53:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3505 [2026-10-20 15:53:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6958 [2026-10-20 15:53:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9564 [2026-10-20 15:53:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8127 [2026-10-20 15:53:02][INFO][training.pretrain_engine] [Ph2|Step 531] loss=5.8775 | ppl=356.92 | aux=0.02510 | lr=3.00e-04 | gnorm=1.234 | tok/s=29,247 | MFU=7.0% | step=143410ms | tokens=0.564B | mem=4.2/32.7GB | cost=$279.04/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 08:59 (137.1h) | Phase3: SKIPPED | SFT: Oct 26 13:44 (141.9h) | DPO (Final): Oct 26 14:18 (142.4h) [2026-10-20 15:53:02][INFO][training.pretrain_engine] Starting global step 532 (accumulating 32 micro-batches)... [2026-10-20 15:53:02][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 15:53:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9172 [2026-10-20 15:53:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1470 [2026-10-20 15:53:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6227 [2026-10-20 15:53:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9588 [2026-10-20 15:53:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0184 [2026-10-20 15:53:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3963 [2026-10-20 15:53:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2868 [2026-10-20 15:53:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3546 [2026-10-20 15:53:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7350 [2026-10-20 15:53:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4476 [2026-10-20 15:53:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5842 [2026-10-20 15:53:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7288 [2026-10-20 15:54:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5902 [2026-10-20 15:54:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7274 [2026-10-20 15:54:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6879 [2026-10-20 15:54:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9548 [2026-10-20 15:54:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4986 [2026-10-20 15:54:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7209 [2026-10-20 15:54:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0591 [2026-10-20 15:54:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.8792 [2026-10-20 15:54:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8522 [2026-10-20 15:54:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0761 [2026-10-20 15:54:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8883 [2026-10-20 15:54:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7218 [2026-10-20 15:55:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9108 [2026-10-20 15:55:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3344 [2026-10-20 15:55:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9823 [2026-10-20 15:55:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6172 [2026-10-20 15:55:22][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,509 | MFU: 1.8% [2026-10-20 15:55:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7456 [2026-10-20 15:55:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3885 [2026-10-20 15:55:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2252 [2026-10-20 15:55:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2953 [2026-10-20 15:55:26][INFO][training.pretrain_engine] [Ph2|Step 532] loss=5.8484 | ppl=346.69 | aux=0.02510 | lr=3.00e-04 | gnorm=0.754 | tok/s=29,246 | MFU=7.0% | step=143416ms | tokens=0.565B | mem=4.2/32.7GB | cost=$279.56/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 09:01 (137.1h) | Phase3: SKIPPED | SFT: Oct 26 13:46 (141.9h) | DPO (Final): Oct 26 14:20 (142.4h) [2026-10-20 15:55:26][INFO][training.pretrain_engine] Starting global step 533 (accumulating 32 micro-batches)... [2026-10-20 15:55:26][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 15:55:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5054 [2026-10-20 15:55:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7469 [2026-10-20 15:55:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9641 [2026-10-20 15:55:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0564 [2026-10-20 15:56:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5904 [2026-10-20 15:56:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4850 [2026-10-20 15:56:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8008 [2026-10-20 15:56:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2669 [2026-10-20 15:56:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6819 [2026-10-20 15:56:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3056 [2026-10-20 15:56:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6207 [2026-10-20 15:56:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5478 [2026-10-20 15:56:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3350 [2026-10-20 15:56:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8598 [2026-10-20 15:56:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3567 [2026-10-20 15:56:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1802 [2026-10-20 15:56:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7896 [2026-10-20 15:56:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6641 [2026-10-20 15:56:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8805 [2026-10-20 15:56:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5479 [2026-10-20 15:57:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0615 [2026-10-20 15:57:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5842 [2026-10-20 15:57:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2801 [2026-10-20 15:57:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0852 [2026-10-20 15:57:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8050 [2026-10-20 15:57:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8146 [2026-10-20 15:57:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3854 [2026-10-20 15:57:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7291 [2026-10-20 15:57:46][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,516 | MFU: 1.8% [2026-10-20 15:57:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=7.0158 [2026-10-20 15:57:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6119 [2026-10-20 15:57:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5748 [2026-10-20 15:57:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5569 [2026-10-20 15:57:50][INFO][training.pretrain_engine] [Ph2|Step 533] loss=5.8186 | ppl=336.49 | aux=0.02510 | lr=3.00e-04 | gnorm=1.055 | tok/s=29,245 | MFU=7.0% | step=143420ms | tokens=0.566B | mem=4.2/32.7GB | cost=$280.09/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 09:03 (137.1h) | Phase3: SKIPPED | SFT: Oct 26 13:48 (141.8h) | DPO (Final): Oct 26 14:22 (142.4h) [2026-10-20 15:57:53][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step533 | Tokens: 0.566B | Spent: $280.10 | Remaining: $119.90 [2026-10-20 15:57:53][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step533 [2026-10-20 15:57:54][INFO][training.pretrain_engine] Starting global step 534 (accumulating 32 micro-batches)... [2026-10-20 15:57:54][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 15:57:54][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 15:57:54][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 15:57:54][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 15:57:54][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step533 to Phase-Technologies/forge-500m-pretrain (path: phase2_step533)... [2026-10-20 15:57:54][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 15:58:04][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 15:58:04][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step533 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 15:58:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7946 [2026-10-20 15:58:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5922 [2026-10-20 15:58:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2378 [2026-10-20 15:58:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6543 [2026-10-20 15:58:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5714 [2026-10-20 15:58:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7528 [2026-10-20 15:58:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0599 [2026-10-20 15:58:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9288 [2026-10-20 15:58:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3150 [2026-10-20 15:58:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0370 [2026-10-20 15:58:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8448 [2026-10-20 15:58:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5767 [2026-10-20 15:59:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3798 [2026-10-20 15:59:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1986 [2026-10-20 15:59:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1485 [2026-10-20 15:59:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0131 [2026-10-20 15:59:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0034 [2026-10-20 15:59:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9449 [2026-10-20 15:59:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7517 [2026-10-20 15:59:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0221 [2026-10-20 15:59:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5405 [2026-10-20 15:59:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3235 [2026-10-20 15:59:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2327 [2026-10-20 15:59:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6371 [2026-10-20 15:59:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3596 [2026-10-20 15:59:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0163 [2026-10-20 15:59:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8542 [2026-10-20 15:59:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2366 [2026-10-20 16:00:13][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,529 | MFU: 1.8% [2026-10-20 16:00:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2549 [2026-10-20 16:00:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2679 [2026-10-20 16:00:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4031 [2026-10-20 16:00:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1600 [2026-10-20 16:00:17][INFO][training.pretrain_engine] [Ph2|Step 534] loss=5.8039 | ppl=331.58 | aux=0.02509 | lr=3.00e-04 | gnorm=0.789 | tok/s=29,246 | MFU=7.0% | step=143417ms | tokens=0.567B | mem=4.2/32.7GB | cost=$280.63/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 09:05 (137.1h) | Phase3: SKIPPED | SFT: Oct 26 13:50 (141.8h) | DPO (Final): Oct 26 14:24 (142.4h) [2026-10-20 16:00:17][INFO][training.pretrain_engine] Starting global step 535 (accumulating 32 micro-batches)... [2026-10-20 16:00:17][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 16:00:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9908 [2026-10-20 16:00:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7950 [2026-10-20 16:00:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5518 [2026-10-20 16:00:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0185 [2026-10-20 16:00:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0719 [2026-10-20 16:00:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5538 [2026-10-20 16:00:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7469 [2026-10-20 16:00:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8056 [2026-10-20 16:01:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9310 [2026-10-20 16:01:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7737 [2026-10-20 16:01:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7614 [2026-10-20 16:01:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0518 [2026-10-20 16:01:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9546 [2026-10-20 16:01:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5199 [2026-10-20 16:01:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6961 [2026-10-20 16:01:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9041 [2026-10-20 16:01:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7696 [2026-10-20 16:01:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9501 [2026-10-20 16:01:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9511 [2026-10-20 16:01:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5044 [2026-10-20 16:02:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7028 [2026-10-20 16:02:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4945 [2026-10-20 16:02:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7076 [2026-10-20 16:02:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7916 [2026-10-20 16:02:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8103 [2026-10-20 16:02:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7803 [2026-10-20 16:02:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8078 [2026-10-20 16:02:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0946 [2026-10-20 16:02:37][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,499 | MFU: 1.8% [2026-10-20 16:02:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8545 [2026-10-20 16:02:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8593 [2026-10-20 16:02:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0129 [2026-10-20 16:02:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3709 [2026-10-20 16:02:42][INFO][training.pretrain_engine] [Ph2|Step 535] loss=5.7608 | ppl=317.62 | aux=0.02509 | lr=3.00e-04 | gnorm=1.172 | tok/s=29,244 | MFU=7.0% | step=143425ms | tokens=0.568B | mem=4.2/32.7GB | cost=$281.16/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 09:07 (137.1h) | Phase3: SKIPPED | SFT: Oct 26 13:52 (141.8h) | DPO (Final): Oct 26 14:26 (142.4h) [2026-10-20 16:02:42][INFO][training.pretrain_engine] Starting global step 536 (accumulating 32 micro-batches)... [2026-10-20 16:02:42][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 16:03:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5717 [2026-10-20 16:03:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9372 [2026-10-20 16:03:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5119 [2026-10-20 16:03:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.8751 [2026-10-20 16:03:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8277 [2026-10-20 16:03:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3527 [2026-10-20 16:03:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9473 [2026-10-20 16:03:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5751 [2026-10-20 16:03:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2108 [2026-10-20 16:03:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0922 [2026-10-20 16:03:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4300 [2026-10-20 16:03:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1586 [2026-10-20 16:03:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9991 [2026-10-20 16:03:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0867 [2026-10-20 16:03:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8843 [2026-10-20 16:03:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5978 [2026-10-20 16:04:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7264 [2026-10-20 16:04:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6411 [2026-10-20 16:04:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6449 [2026-10-20 16:04:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6906 [2026-10-20 16:04:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7955 [2026-10-20 16:04:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2176 [2026-10-20 16:04:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9846 [2026-10-20 16:04:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9453 [2026-10-20 16:04:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0101 [2026-10-20 16:04:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6631 [2026-10-20 16:04:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4820 [2026-10-20 16:04:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5124 [2026-10-20 16:05:01][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,543 | MFU: 1.8% [2026-10-20 16:05:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4790 [2026-10-20 16:05:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5621 [2026-10-20 16:05:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0539 [2026-10-20 16:05:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4688 [2026-10-20 16:05:05][INFO][training.pretrain_engine] [Ph2|Step 536] loss=5.9256 | ppl=374.51 | aux=0.02510 | lr=3.00e-04 | gnorm=0.863 | tok/s=29,244 | MFU=7.0% | step=143424ms | tokens=0.569B | mem=4.2/32.7GB | cost=$281.68/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 09:09 (137.1h) | Phase3: SKIPPED | SFT: Oct 26 13:54 (141.8h) | DPO (Final): Oct 26 14:28 (142.4h) [2026-10-20 16:05:05][INFO][training.pretrain_engine] Starting global step 537 (accumulating 32 micro-batches)... [2026-10-20 16:05:05][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 16:05:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6119 [2026-10-20 16:05:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9559 [2026-10-20 16:05:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1134 [2026-10-20 16:05:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5828 [2026-10-20 16:05:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4796 [2026-10-20 16:05:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3886 [2026-10-20 16:05:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4720 [2026-10-20 16:05:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6030 [2026-10-20 16:05:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8653 [2026-10-20 16:05:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5811 [2026-10-20 16:05:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8017 [2026-10-20 16:05:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4750 [2026-10-20 16:06:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2873 [2026-10-20 16:06:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9364 [2026-10-20 16:06:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4879 [2026-10-20 16:06:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1820 [2026-10-20 16:06:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9702 [2026-10-20 16:06:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4583 [2026-10-20 16:06:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5258 [2026-10-20 16:06:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9951 [2026-10-20 16:06:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0378 [2026-10-20 16:06:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5266 [2026-10-20 16:06:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5552 [2026-10-20 16:06:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5334 [2026-10-20 16:07:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9764 [2026-10-20 16:07:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2542 [2026-10-20 16:07:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9515 [2026-10-20 16:07:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4582 [2026-10-20 16:07:24][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,541 | MFU: 1.8% [2026-10-20 16:07:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5673 [2026-10-20 16:07:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7589 [2026-10-20 16:07:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1086 [2026-10-20 16:07:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7168 [2026-10-20 16:07:29][INFO][training.pretrain_engine] [Ph2|Step 537] loss=5.7727 | ppl=321.41 | aux=0.02511 | lr=3.00e-04 | gnorm=1.008 | tok/s=29,244 | MFU=7.0% | step=143425ms | tokens=0.570B | mem=4.2/32.7GB | cost=$282.21/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 09:11 (137.1h) | Phase3: SKIPPED | SFT: Oct 26 13:56 (141.8h) | DPO (Final): Oct 26 14:30 (142.4h) [2026-10-20 16:07:29][INFO][training.pretrain_engine] Starting global step 538 (accumulating 32 micro-batches)... [2026-10-20 16:07:29][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 16:07:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7286 [2026-10-20 16:07:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4343 [2026-10-20 16:07:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9216 [2026-10-20 16:07:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7526 [2026-10-20 16:08:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5844 [2026-10-20 16:08:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5607 [2026-10-20 16:08:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4723 [2026-10-20 16:08:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4828 [2026-10-20 16:08:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0321 [2026-10-20 16:08:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9149 [2026-10-20 16:08:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9535 [2026-10-20 16:08:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7719 [2026-10-20 16:08:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6036 [2026-10-20 16:08:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1828 [2026-10-20 16:08:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0008 [2026-10-20 16:08:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8512 [2026-10-20 16:08:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0140 [2026-10-20 16:08:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5540 [2026-10-20 16:08:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7136 [2026-10-20 16:08:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4522 [2026-10-20 16:09:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1647 [2026-10-20 16:09:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6827 [2026-10-20 16:09:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3244 [2026-10-20 16:09:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6214 [2026-10-20 16:09:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8803 [2026-10-20 16:09:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5333 [2026-10-20 16:09:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6190 [2026-10-20 16:09:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0209 [2026-10-20 16:09:48][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,537 | MFU: 1.8% [2026-10-20 16:09:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9324 [2026-10-20 16:09:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3497 [2026-10-20 16:09:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3952 [2026-10-20 16:09:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5893 [2026-10-20 16:09:53][INFO][training.pretrain_engine] [Ph2|Step 538] loss=5.7539 | ppl=315.42 | aux=0.02511 | lr=3.00e-04 | gnorm=0.789 | tok/s=29,244 | MFU=7.0% | step=143426ms | tokens=0.571B | mem=4.2/32.7GB | cost=$282.73/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 09:13 (137.1h) | Phase3: SKIPPED | SFT: Oct 26 13:58 (141.8h) | DPO (Final): Oct 26 14:32 (142.4h) [2026-10-20 16:09:53][INFO][training.pretrain_engine] Starting global step 539 (accumulating 32 micro-batches)... [2026-10-20 16:09:53][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 16:10:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8351 [2026-10-20 16:10:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2529 [2026-10-20 16:10:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4538 [2026-10-20 16:10:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5497 [2026-10-20 16:10:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9026 [2026-10-20 16:10:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9122 [2026-10-20 16:10:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7381 [2026-10-20 16:10:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9016 [2026-10-20 16:10:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0613 [2026-10-20 16:10:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0333 [2026-10-20 16:10:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8592 [2026-10-20 16:10:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8479 [2026-10-20 16:11:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3966 [2026-10-20 16:11:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5713 [2026-10-20 16:11:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2913 [2026-10-20 16:11:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3264 [2026-10-20 16:11:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6972 [2026-10-20 16:11:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4726 [2026-10-20 16:11:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5661 [2026-10-20 16:11:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9558 [2026-10-20 16:11:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0431 [2026-10-20 16:11:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2551 [2026-10-20 16:11:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1736 [2026-10-20 16:11:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3232 [2026-10-20 16:11:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0590 [2026-10-20 16:11:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8873 [2026-10-20 16:11:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5326 [2026-10-20 16:11:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7557 [2026-10-20 16:12:11][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,558 | MFU: 1.8% [2026-10-20 16:12:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7141 [2026-10-20 16:12:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0496 [2026-10-20 16:12:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8114 [2026-10-20 16:12:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9553 [2026-10-20 16:12:16][INFO][training.pretrain_engine] [Ph2|Step 539] loss=5.7545 | ppl=315.61 | aux=0.02510 | lr=3.00e-04 | gnorm=0.902 | tok/s=29,245 | MFU=7.0% | step=143422ms | tokens=0.573B | mem=4.2/32.7GB | cost=$283.26/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 09:14 (137.0h) | Phase3: SKIPPED | SFT: Oct 26 13:59 (141.8h) | DPO (Final): Oct 26 14:33 (142.4h) [2026-10-20 16:12:16][INFO][training.pretrain_engine] Starting global step 540 (accumulating 32 micro-batches)... [2026-10-20 16:12:16][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 16:12:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8519 [2026-10-20 16:12:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6747 [2026-10-20 16:12:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6418 [2026-10-20 16:12:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4509 [2026-10-20 16:12:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6787 [2026-10-20 16:12:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8039 [2026-10-20 16:12:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7406 [2026-10-20 16:12:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3039 [2026-10-20 16:13:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8352 [2026-10-20 16:13:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6714 [2026-10-20 16:13:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1624 [2026-10-20 16:13:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.8314 [2026-10-20 16:13:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6207 [2026-10-20 16:13:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9467 [2026-10-20 16:13:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5932 [2026-10-20 16:13:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8864 [2026-10-20 16:13:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7711 [2026-10-20 16:13:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6673 [2026-10-20 16:13:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3751 [2026-10-20 16:13:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5629 [2026-10-20 16:14:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5610 [2026-10-20 16:14:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0002 [2026-10-20 16:14:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6308 [2026-10-20 16:14:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8763 [2026-10-20 16:14:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8169 [2026-10-20 16:14:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1630 [2026-10-20 16:14:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5443 [2026-10-20 16:14:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5275 [2026-10-20 16:14:34][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,580 | MFU: 1.8% [2026-10-20 16:14:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7026 [2026-10-20 16:14:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9351 [2026-10-20 16:14:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9345 [2026-10-20 16:14:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5657 [2026-10-20 16:14:39][INFO][training.pretrain_engine] [Ph2|Step 540] loss=5.7544 | ppl=315.58 | aux=0.02509 | lr=3.00e-04 | gnorm=0.809 | tok/s=29,247 | MFU=7.0% | step=143410ms | tokens=0.574B | mem=4.2/32.7GB | cost=$283.78/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 09:15 (137.0h) | Phase3: SKIPPED | SFT: Oct 26 14:00 (141.8h) | DPO (Final): Oct 26 14:34 (142.3h) [2026-10-20 16:14:39][INFO][training.pretrain_engine] Starting global step 541 (accumulating 32 micro-batches)... [2026-10-20 16:14:39][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 16:14:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9534 [2026-10-20 16:14:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6128 [2026-10-20 16:14:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8036 [2026-10-20 16:14:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8422 [2026-10-20 16:15:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7528 [2026-10-20 16:15:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6093 [2026-10-20 16:15:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0679 [2026-10-20 16:15:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1210 [2026-10-20 16:15:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9611 [2026-10-20 16:15:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9076 [2026-10-20 16:15:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8706 [2026-10-20 16:15:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3984 [2026-10-20 16:15:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8570 [2026-10-20 16:15:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0419 [2026-10-20 16:15:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6959 [2026-10-20 16:15:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4244 [2026-10-20 16:16:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9485 [2026-10-20 16:16:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9639 [2026-10-20 16:16:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7089 [2026-10-20 16:16:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6603 [2026-10-20 16:16:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4854 [2026-10-20 16:16:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7791 [2026-10-20 16:16:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6946 [2026-10-20 16:16:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0352 [2026-10-20 16:16:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8066 [2026-10-20 16:16:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9954 [2026-10-20 16:16:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9189 [2026-10-20 16:16:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6855 [2026-10-20 16:16:58][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,553 | MFU: 1.8% [2026-10-20 16:17:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9935 [2026-10-20 16:17:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4907 [2026-10-20 16:17:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8006 [2026-10-20 16:17:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0465 [2026-10-20 16:17:03][INFO][training.pretrain_engine] [Ph2|Step 541] loss=5.8604 | ppl=350.85 | aux=0.02509 | lr=3.00e-04 | gnorm=0.965 | tok/s=29,247 | MFU=7.0% | step=143409ms | tokens=0.575B | mem=4.2/32.7GB | cost=$284.30/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 09:17 (137.0h) | Phase3: SKIPPED | SFT: Oct 26 14:02 (141.8h) | DPO (Final): Oct 26 14:36 (142.3h) [2026-10-20 16:17:03][INFO][training.pretrain_engine] Starting global step 542 (accumulating 32 micro-batches)... [2026-10-20 16:17:03][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 16:17:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5211 [2026-10-20 16:17:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3351 [2026-10-20 16:17:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1592 [2026-10-20 16:17:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0734 [2026-10-20 16:17:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6869 [2026-10-20 16:17:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3880 [2026-10-20 16:17:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9176 [2026-10-20 16:17:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6033 [2026-10-20 16:17:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8208 [2026-10-20 16:17:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5845 [2026-10-20 16:17:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6981 [2026-10-20 16:17:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2675 [2026-10-20 16:18:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9487 [2026-10-20 16:18:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8163 [2026-10-20 16:18:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5542 [2026-10-20 16:18:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8994 [2026-10-20 16:18:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5104 [2026-10-20 16:18:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1344 [2026-10-20 16:18:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4092 [2026-10-20 16:18:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9425 [2026-10-20 16:18:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8957 [2026-10-20 16:18:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4129 [2026-10-20 16:18:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8027 [2026-10-20 16:18:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6569 [2026-10-20 16:19:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7942 [2026-10-20 16:19:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8240 [2026-10-20 16:19:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1693 [2026-10-20 16:19:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4897 [2026-10-20 16:19:22][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,514 | MFU: 1.8% [2026-10-20 16:19:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5923 [2026-10-20 16:19:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8001 [2026-10-20 16:19:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6126 [2026-10-20 16:19:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4158 [2026-10-20 16:19:27][INFO][training.pretrain_engine] [Ph2|Step 542] loss=5.6769 | ppl=292.04 | aux=0.02509 | lr=3.00e-04 | gnorm=1.266 | tok/s=29,247 | MFU=7.0% | step=143410ms | tokens=0.576B | mem=4.2/32.7GB | cost=$284.83/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 09:19 (137.0h) | Phase3: SKIPPED | SFT: Oct 26 14:04 (141.7h) | DPO (Final): Oct 26 14:38 (142.3h) [2026-10-20 16:19:30][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step542 | Tokens: 0.576B | Spent: $284.84 | Remaining: $115.16 [2026-10-20 16:19:30][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step542 [2026-10-20 16:19:30][INFO][training.pretrain_engine] Starting global step 543 (accumulating 32 micro-batches)... [2026-10-20 16:19:30][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 16:19:30][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 16:19:30][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 16:19:30][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 16:19:30][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step542 to Phase-Technologies/forge-500m-pretrain (path: phase2_step542)... [2026-10-20 16:19:31][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 16:19:41][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 16:19:41][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step542 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 16:19:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0186 [2026-10-20 16:19:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9254 [2026-10-20 16:19:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3549 [2026-10-20 16:19:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2124 [2026-10-20 16:20:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5551 [2026-10-20 16:20:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8949 [2026-10-20 16:20:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4587 [2026-10-20 16:20:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1112 [2026-10-20 16:20:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0046 [2026-10-20 16:20:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7323 [2026-10-20 16:20:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4102 [2026-10-20 16:20:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4459 [2026-10-20 16:20:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9017 [2026-10-20 16:20:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7295 [2026-10-20 16:20:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7924 [2026-10-20 16:20:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8225 [2026-10-20 16:20:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9523 [2026-10-20 16:20:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8065 [2026-10-20 16:20:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2295 [2026-10-20 16:21:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0938 [2026-10-20 16:21:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0043 [2026-10-20 16:21:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5516 [2026-10-20 16:21:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1518 [2026-10-20 16:21:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1810 [2026-10-20 16:21:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4967 [2026-10-20 16:21:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2174 [2026-10-20 16:21:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1642 [2026-10-20 16:21:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5223 [2026-10-20 16:21:50][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,530 | MFU: 1.8% [2026-10-20 16:21:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4289 [2026-10-20 16:21:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3020 [2026-10-20 16:21:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9497 [2026-10-20 16:21:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3057 [2026-10-20 16:21:54][INFO][training.pretrain_engine] [Ph2|Step 543] loss=5.6665 | ppl=289.03 | aux=0.02508 | lr=3.00e-04 | gnorm=0.988 | tok/s=29,247 | MFU=7.0% | step=143409ms | tokens=0.577B | mem=4.2/32.7GB | cost=$285.37/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 09:21 (137.0h) | Phase3: SKIPPED | SFT: Oct 26 14:06 (141.7h) | DPO (Final): Oct 26 14:40 (142.3h) [2026-10-20 16:21:54][INFO][training.pretrain_engine] Starting global step 544 (accumulating 32 micro-batches)... [2026-10-20 16:21:54][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 16:22:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9192 [2026-10-20 16:22:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8837 [2026-10-20 16:22:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2677 [2026-10-20 16:22:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8760 [2026-10-20 16:22:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1093 [2026-10-20 16:22:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9136 [2026-10-20 16:22:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1490 [2026-10-20 16:22:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6807 [2026-10-20 16:22:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8719 [2026-10-20 16:22:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5799 [2026-10-20 16:22:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8402 [2026-10-20 16:22:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7287 [2026-10-20 16:23:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8297 [2026-10-20 16:23:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1816 [2026-10-20 16:23:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8742 [2026-10-20 16:23:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7040 [2026-10-20 16:23:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7639 [2026-10-20 16:23:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9293 [2026-10-20 16:23:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8642 [2026-10-20 16:23:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6210 [2026-10-20 16:23:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1052 [2026-10-20 16:23:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7020 [2026-10-20 16:23:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7786 [2026-10-20 16:23:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6920 [2026-10-20 16:23:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7364 [2026-10-20 16:24:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7852 [2026-10-20 16:24:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9235 [2026-10-20 16:24:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7584 [2026-10-20 16:24:13][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,547 | MFU: 1.8% [2026-10-20 16:24:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8977 [2026-10-20 16:24:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0854 [2026-10-20 16:24:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8861 [2026-10-20 16:24:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3391 [2026-10-20 16:24:18][INFO][training.pretrain_engine] [Ph2|Step 544] loss=5.8599 | ppl=350.68 | aux=0.02509 | lr=3.00e-04 | gnorm=0.762 | tok/s=29,248 | MFU=7.0% | step=143406ms | tokens=0.578B | mem=4.2/32.7GB | cost=$285.89/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 09:22 (137.0h) | Phase3: SKIPPED | SFT: Oct 26 14:07 (141.7h) | DPO (Final): Oct 26 14:41 (142.3h) [2026-10-20 16:24:18][INFO][training.pretrain_engine] Starting global step 545 (accumulating 32 micro-batches)... [2026-10-20 16:24:18][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 16:24:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8376 [2026-10-20 16:24:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7082 [2026-10-20 16:24:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7421 [2026-10-20 16:24:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1116 [2026-10-20 16:24:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1410 [2026-10-20 16:24:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3978 [2026-10-20 16:24:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0712 [2026-10-20 16:24:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6248 [2026-10-20 16:25:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6797 [2026-10-20 16:25:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4895 [2026-10-20 16:25:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3522 [2026-10-20 16:25:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9473 [2026-10-20 16:25:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8794 [2026-10-20 16:25:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5173 [2026-10-20 16:25:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9300 [2026-10-20 16:25:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0716 [2026-10-20 16:25:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1970 [2026-10-20 16:25:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6658 [2026-10-20 16:25:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9716 [2026-10-20 16:25:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8308 [2026-10-20 16:26:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4514 [2026-10-20 16:26:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4533 [2026-10-20 16:26:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8841 [2026-10-20 16:26:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7686 [2026-10-20 16:26:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5966 [2026-10-20 16:26:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8803 [2026-10-20 16:26:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3636 [2026-10-20 16:26:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7229 [2026-10-20 16:26:37][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,525 | MFU: 1.8% [2026-10-20 16:26:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8261 [2026-10-20 16:26:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2444 [2026-10-20 16:26:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8545 [2026-10-20 16:26:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0632 [2026-10-20 16:26:42][INFO][training.pretrain_engine] [Ph2|Step 545] loss=5.8513 | ppl=347.70 | aux=0.02510 | lr=3.00e-04 | gnorm=0.719 | tok/s=29,247 | MFU=7.0% | step=143408ms | tokens=0.579B | mem=4.2/32.7GB | cost=$286.42/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 09:24 (137.0h) | Phase3: SKIPPED | SFT: Oct 26 14:09 (141.7h) | DPO (Final): Oct 26 14:43 (142.3h) [2026-10-20 16:26:42][INFO][training.pretrain_engine] Starting global step 546 (accumulating 32 micro-batches)... [2026-10-20 16:26:42][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 16:27:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2319 [2026-10-20 16:27:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9606 [2026-10-20 16:27:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8297 [2026-10-20 16:27:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0481 [2026-10-20 16:27:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1356 [2026-10-20 16:27:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6108 [2026-10-20 16:27:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9638 [2026-10-20 16:27:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7344 [2026-10-20 16:27:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6222 [2026-10-20 16:27:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5507 [2026-10-20 16:27:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6850 [2026-10-20 16:27:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5565 [2026-10-20 16:27:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5156 [2026-10-20 16:27:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9787 [2026-10-20 16:27:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9332 [2026-10-20 16:27:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8455 [2026-10-20 16:28:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9330 [2026-10-20 16:28:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0342 [2026-10-20 16:28:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8265 [2026-10-20 16:28:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9542 [2026-10-20 16:28:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3972 [2026-10-20 16:28:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7938 [2026-10-20 16:28:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2743 [2026-10-20 16:28:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2534 [2026-10-20 16:28:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6920 [2026-10-20 16:28:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4501 [2026-10-20 16:28:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7872 [2026-10-20 16:28:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9700 [2026-10-20 16:29:00][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,579 | MFU: 1.8% [2026-10-20 16:29:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2269 [2026-10-20 16:29:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5434 [2026-10-20 16:29:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3009 [2026-10-20 16:29:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8916 [2026-10-20 16:29:05][INFO][training.pretrain_engine] [Ph2|Step 546] loss=5.7976 | ppl=329.52 | aux=0.02509 | lr=3.00e-04 | gnorm=0.824 | tok/s=29,248 | MFU=7.0% | step=143404ms | tokens=0.580B | mem=4.2/32.7GB | cost=$286.94/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 09:26 (137.0h) | Phase3: SKIPPED | SFT: Oct 26 14:11 (141.7h) | DPO (Final): Oct 26 14:45 (142.3h) [2026-10-20 16:29:05][INFO][training.pretrain_engine] Starting global step 547 (accumulating 32 micro-batches)... [2026-10-20 16:29:05][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 16:29:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9496 [2026-10-20 16:29:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7701 [2026-10-20 16:29:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0206 [2026-10-20 16:29:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7763 [2026-10-20 16:29:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0921 [2026-10-20 16:29:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4876 [2026-10-20 16:29:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3328 [2026-10-20 16:29:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7314 [2026-10-20 16:29:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5110 [2026-10-20 16:29:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6837 [2026-10-20 16:29:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8402 [2026-10-20 16:29:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2603 [2026-10-20 16:30:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9607 [2026-10-20 16:30:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7989 [2026-10-20 16:30:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0230 [2026-10-20 16:30:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9981 [2026-10-20 16:30:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7855 [2026-10-20 16:30:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6923 [2026-10-20 16:30:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5245 [2026-10-20 16:30:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0259 [2026-10-20 16:30:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9279 [2026-10-20 16:30:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7107 [2026-10-20 16:30:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1084 [2026-10-20 16:30:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9160 [2026-10-20 16:31:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0743 [2026-10-20 16:31:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8691 [2026-10-20 16:31:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2073 [2026-10-20 16:31:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9345 [2026-10-20 16:31:24][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,575 | MFU: 1.8% [2026-10-20 16:31:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8727 [2026-10-20 16:31:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9075 [2026-10-20 16:31:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7612 [2026-10-20 16:31:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7882 [2026-10-20 16:31:28][INFO][training.pretrain_engine] [Ph2|Step 547] loss=5.8664 | ppl=352.99 | aux=0.02509 | lr=3.00e-04 | gnorm=0.902 | tok/s=29,250 | MFU=7.0% | step=143396ms | tokens=0.581B | mem=4.2/32.7GB | cost=$287.47/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 09:27 (136.9h) | Phase3: SKIPPED | SFT: Oct 26 14:12 (141.7h) | DPO (Final): Oct 26 14:46 (142.3h) [2026-10-20 16:31:28][INFO][training.pretrain_engine] Starting global step 548 (accumulating 32 micro-batches)... [2026-10-20 16:31:28][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 16:31:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2293 [2026-10-20 16:31:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4985 [2026-10-20 16:31:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5033 [2026-10-20 16:31:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3386 [2026-10-20 16:32:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2763 [2026-10-20 16:32:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7932 [2026-10-20 16:32:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0985 [2026-10-20 16:32:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3944 [2026-10-20 16:32:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0599 [2026-10-20 16:32:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1121 [2026-10-20 16:32:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4084 [2026-10-20 16:32:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2297 [2026-10-20 16:32:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7276 [2026-10-20 16:32:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0539 [2026-10-20 16:32:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5130 [2026-10-20 16:32:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7333 [2026-10-20 16:32:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7099 [2026-10-20 16:32:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2238 [2026-10-20 16:32:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9705 [2026-10-20 16:32:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6144 [2026-10-20 16:33:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3391 [2026-10-20 16:33:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4492 [2026-10-20 16:33:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8662 [2026-10-20 16:33:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2098 [2026-10-20 16:33:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9299 [2026-10-20 16:33:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7670 [2026-10-20 16:33:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6324 [2026-10-20 16:33:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8749 [2026-10-20 16:33:46][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,589 | MFU: 1.8% [2026-10-20 16:33:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7171 [2026-10-20 16:33:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6911 [2026-10-20 16:33:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4562 [2026-10-20 16:33:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2707 [2026-10-20 16:33:51][INFO][training.pretrain_engine] [Ph2|Step 548] loss=5.7255 | ppl=306.59 | aux=0.02510 | lr=3.00e-04 | gnorm=1.117 | tok/s=29,252 | MFU=7.0% | step=143387ms | tokens=0.582B | mem=4.2/32.7GB | cost=$287.99/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 09:28 (136.9h) | Phase3: SKIPPED | SFT: Oct 26 14:13 (141.7h) | DPO (Final): Oct 26 14:47 (142.2h) [2026-10-20 16:33:51][INFO][training.pretrain_engine] Starting global step 549 (accumulating 32 micro-batches)... [2026-10-20 16:33:51][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 16:34:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6095 [2026-10-20 16:34:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9920 [2026-10-20 16:34:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6777 [2026-10-20 16:34:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0783 [2026-10-20 16:34:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8571 [2026-10-20 16:34:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7049 [2026-10-20 16:34:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6796 [2026-10-20 16:34:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8003 [2026-10-20 16:34:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.7629 [2026-10-20 16:34:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1011 [2026-10-20 16:34:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4947 [2026-10-20 16:34:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7523 [2026-10-20 16:35:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0846 [2026-10-20 16:35:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2351 [2026-10-20 16:35:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3596 [2026-10-20 16:35:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4092 [2026-10-20 16:35:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1854 [2026-10-20 16:35:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8892 [2026-10-20 16:35:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9718 [2026-10-20 16:35:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0374 [2026-10-20 16:35:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3071 [2026-10-20 16:35:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1657 [2026-10-20 16:35:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1078 [2026-10-20 16:35:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4266 [2026-10-20 16:35:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9644 [2026-10-20 16:35:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6685 [2026-10-20 16:35:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8185 [2026-10-20 16:35:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6098 [2026-10-20 16:36:10][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,572 | MFU: 1.8% [2026-10-20 16:36:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6050 [2026-10-20 16:36:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0325 [2026-10-20 16:36:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3011 [2026-10-20 16:36:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6471 [2026-10-20 16:36:14][INFO][training.pretrain_engine] [Ph2|Step 549] loss=5.7500 | ppl=314.18 | aux=0.02509 | lr=3.00e-04 | gnorm=0.809 | tok/s=29,254 | MFU=7.0% | step=143377ms | tokens=0.583B | mem=4.2/32.7GB | cost=$288.51/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 09:30 (136.9h) | Phase3: SKIPPED | SFT: Oct 26 14:14 (141.6h) | DPO (Final): Oct 26 14:49 (142.2h) [2026-10-20 16:36:14][INFO][training.pretrain_engine] Starting global step 550 (accumulating 32 micro-batches)... [2026-10-20 16:36:14][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 16:36:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0091 [2026-10-20 16:36:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3058 [2026-10-20 16:36:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6502 [2026-10-20 16:36:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9952 [2026-10-20 16:36:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0311 [2026-10-20 16:36:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7981 [2026-10-20 16:36:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7684 [2026-10-20 16:36:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1829 [2026-10-20 16:37:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0814 [2026-10-20 16:37:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4444 [2026-10-20 16:37:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5951 [2026-10-20 16:37:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0573 [2026-10-20 16:37:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8185 [2026-10-20 16:37:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5390 [2026-10-20 16:37:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6827 [2026-10-20 16:37:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3613 [2026-10-20 16:37:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4021 [2026-10-20 16:37:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6186 [2026-10-20 16:37:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5544 [2026-10-20 16:37:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9927 [2026-10-20 16:38:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7063 [2026-10-20 16:38:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6387 [2026-10-20 16:38:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5631 [2026-10-20 16:38:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8530 [2026-10-20 16:38:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8926 [2026-10-20 16:38:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9810 [2026-10-20 16:38:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0147 [2026-10-20 16:38:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6301 [2026-10-20 16:38:32][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,587 | MFU: 1.8% [2026-10-20 16:38:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7805 [2026-10-20 16:38:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3737 [2026-10-20 16:38:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8073 [2026-10-20 16:38:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3943 [2026-10-20 16:38:37][INFO][training.pretrain_engine] [Ph2|Step 550] loss=5.7500 | ppl=314.19 | aux=0.02508 | lr=3.00e-04 | gnorm=0.840 | tok/s=29,255 | MFU=7.0% | step=143371ms | tokens=0.584B | mem=4.2/32.7GB | cost=$289.04/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 09:31 (136.9h) | Phase3: SKIPPED | SFT: Oct 26 14:16 (141.6h) | DPO (Final): Oct 26 14:50 (142.2h) [2026-10-20 16:38:37][INFO][training.pretrain_engine] Starting global step 551 (accumulating 32 micro-batches)... [2026-10-20 16:38:37][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 16:38:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7275 [2026-10-20 16:38:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1570 [2026-10-20 16:38:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0642 [2026-10-20 16:38:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8024 [2026-10-20 16:39:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3116 [2026-10-20 16:39:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9608 [2026-10-20 16:39:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5239 [2026-10-20 16:39:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5512 [2026-10-20 16:39:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7305 [2026-10-20 16:39:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6842 [2026-10-20 16:39:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1079 [2026-10-20 16:39:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9729 [2026-10-20 16:39:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9682 [2026-10-20 16:39:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1622 [2026-10-20 16:39:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0920 [2026-10-20 16:39:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3175 [2026-10-20 16:40:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1417 [2026-10-20 16:40:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6232 [2026-10-20 16:40:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2940 [2026-10-20 16:40:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5778 [2026-10-20 16:40:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9173 [2026-10-20 16:40:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8969 [2026-10-20 16:40:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6060 [2026-10-20 16:40:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6906 [2026-10-20 16:40:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2393 [2026-10-20 16:40:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6914 [2026-10-20 16:40:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7466 [2026-10-20 16:40:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8928 [2026-10-20 16:40:55][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,591 | MFU: 1.8% [2026-10-20 16:41:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6411 [2026-10-20 16:41:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6387 [2026-10-20 16:41:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2687 [2026-10-20 16:41:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7502 [2026-10-20 16:41:00][INFO][training.pretrain_engine] [Ph2|Step 551] loss=5.7916 | ppl=327.55 | aux=0.02510 | lr=3.00e-04 | gnorm=0.742 | tok/s=29,255 | MFU=7.0% | step=143370ms | tokens=0.585B | mem=4.2/32.7GB | cost=$289.56/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 09:33 (136.9h) | Phase3: SKIPPED | SFT: Oct 26 14:18 (141.6h) | DPO (Final): Oct 26 14:52 (142.2h) [2026-10-20 16:41:04][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step551 | Tokens: 0.585B | Spent: $289.57 | Remaining: $110.43 [2026-10-20 16:41:04][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step551 [2026-10-20 16:41:04][INFO][training.pretrain_engine] Starting global step 552 (accumulating 32 micro-batches)... [2026-10-20 16:41:04][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 16:41:04][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 16:41:04][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 16:41:04][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 16:41:04][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step551 to Phase-Technologies/forge-500m-pretrain (path: phase2_step551)... [2026-10-20 16:41:04][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 16:41:13][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 16:41:13][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step551 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 16:41:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6548 [2026-10-20 16:41:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9592 [2026-10-20 16:41:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7020 [2026-10-20 16:41:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7112 [2026-10-20 16:41:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0790 [2026-10-20 16:41:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0680 [2026-10-20 16:41:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4441 [2026-10-20 16:41:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9728 [2026-10-20 16:41:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5511 [2026-10-20 16:41:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0971 [2026-10-20 16:41:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6897 [2026-10-20 16:41:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7237 [2026-10-20 16:42:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7897 [2026-10-20 16:42:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9354 [2026-10-20 16:42:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9314 [2026-10-20 16:42:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6197 [2026-10-20 16:42:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4278 [2026-10-20 16:42:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7106 [2026-10-20 16:42:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0620 [2026-10-20 16:42:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5798 [2026-10-20 16:42:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9812 [2026-10-20 16:42:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2831 [2026-10-20 16:42:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6498 [2026-10-20 16:42:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1448 [2026-10-20 16:43:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9633 [2026-10-20 16:43:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9059 [2026-10-20 16:43:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4965 [2026-10-20 16:43:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1051 [2026-10-20 16:43:23][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,543 | MFU: 1.8% [2026-10-20 16:43:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7353 [2026-10-20 16:43:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6410 [2026-10-20 16:43:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5248 [2026-10-20 16:43:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5980 [2026-10-20 16:43:27][INFO][training.pretrain_engine] [Ph2|Step 552] loss=5.7763 | ppl=322.56 | aux=0.02510 | lr=3.00e-04 | gnorm=0.625 | tok/s=29,255 | MFU=7.0% | step=143372ms | tokens=0.586B | mem=4.2/32.7GB | cost=$290.10/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 09:35 (136.9h) | Phase3: SKIPPED | SFT: Oct 26 14:20 (141.6h) | DPO (Final): Oct 26 14:54 (142.2h) [2026-10-20 16:43:27][INFO][training.pretrain_engine] Starting global step 553 (accumulating 32 micro-batches)... [2026-10-20 16:43:27][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 16:43:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7625 [2026-10-20 16:43:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7544 [2026-10-20 16:43:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9331 [2026-10-20 16:43:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9508 [2026-10-20 16:44:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4249 [2026-10-20 16:44:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5589 [2026-10-20 16:44:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6277 [2026-10-20 16:44:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5988 [2026-10-20 16:44:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0462 [2026-10-20 16:44:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8795 [2026-10-20 16:44:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3761 [2026-10-20 16:44:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9161 [2026-10-20 16:44:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1996 [2026-10-20 16:44:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4084 [2026-10-20 16:44:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8470 [2026-10-20 16:44:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8095 [2026-10-20 16:44:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6403 [2026-10-20 16:44:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7247 [2026-10-20 16:44:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7047 [2026-10-20 16:44:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5074 [2026-10-20 16:45:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9235 [2026-10-20 16:45:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6769 [2026-10-20 16:45:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5446 [2026-10-20 16:45:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4063 [2026-10-20 16:45:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6234 [2026-10-20 16:45:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6621 [2026-10-20 16:45:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8725 [2026-10-20 16:45:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0370 [2026-10-20 16:45:46][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,580 | MFU: 1.8% [2026-10-20 16:45:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1710 [2026-10-20 16:45:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9500 [2026-10-20 16:45:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6569 [2026-10-20 16:45:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9754 [2026-10-20 16:45:50][INFO][training.pretrain_engine] [Ph2|Step 553] loss=5.8095 | ppl=333.44 | aux=0.02508 | lr=3.00e-04 | gnorm=0.816 | tok/s=29,256 | MFU=7.0% | step=143364ms | tokens=0.587B | mem=4.2/32.7GB | cost=$290.62/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 09:36 (136.8h) | Phase3: SKIPPED | SFT: Oct 26 14:21 (141.6h) | DPO (Final): Oct 26 14:55 (142.2h) [2026-10-20 16:45:50][INFO][training.pretrain_engine] Starting global step 554 (accumulating 32 micro-batches)... [2026-10-20 16:45:50][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 16:46:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6024 [2026-10-20 16:46:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6995 [2026-10-20 16:46:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6663 [2026-10-20 16:46:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1825 [2026-10-20 16:46:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7698 [2026-10-20 16:46:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9672 [2026-10-20 16:46:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8617 [2026-10-20 16:46:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3097 [2026-10-20 16:46:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3357 [2026-10-20 16:46:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4637 [2026-10-20 16:46:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9661 [2026-10-20 16:46:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3850 [2026-10-20 16:47:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7933 [2026-10-20 16:47:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6944 [2026-10-20 16:47:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6417 [2026-10-20 16:47:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8459 [2026-10-20 16:47:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7679 [2026-10-20 16:47:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6976 [2026-10-20 16:47:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0013 [2026-10-20 16:47:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6205 [2026-10-20 16:47:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0221 [2026-10-20 16:47:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8204 [2026-10-20 16:47:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5694 [2026-10-20 16:47:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6628 [2026-10-20 16:47:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8515 [2026-10-20 16:47:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3498 [2026-10-20 16:47:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5641 [2026-10-20 16:47:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6288 [2026-10-20 16:48:08][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,616 | MFU: 1.8% [2026-10-20 16:48:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6314 [2026-10-20 16:48:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4962 [2026-10-20 16:48:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9306 [2026-10-20 16:48:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7357 [2026-10-20 16:48:13][INFO][training.pretrain_engine] [Ph2|Step 554] loss=5.8470 | ppl=346.19 | aux=0.02509 | lr=3.00e-04 | gnorm=0.695 | tok/s=29,258 | MFU=7.0% | step=143356ms | tokens=0.588B | mem=4.2/32.7GB | cost=$291.14/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 09:37 (136.8h) | Phase3: SKIPPED | SFT: Oct 26 14:22 (141.6h) | DPO (Final): Oct 26 14:56 (142.1h) [2026-10-20 16:48:13][INFO][training.pretrain_engine] Starting global step 555 (accumulating 32 micro-batches)... [2026-10-20 16:48:13][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 16:48:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9652 [2026-10-20 16:48:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0281 [2026-10-20 16:48:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5950 [2026-10-20 16:48:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7433 [2026-10-20 16:48:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4785 [2026-10-20 16:48:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1699 [2026-10-20 16:48:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1634 [2026-10-20 16:48:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3773 [2026-10-20 16:49:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7903 [2026-10-20 16:49:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0339 [2026-10-20 16:49:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0425 [2026-10-20 16:49:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7480 [2026-10-20 16:49:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6819 [2026-10-20 16:49:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7559 [2026-10-20 16:49:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5238 [2026-10-20 16:49:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5911 [2026-10-20 16:49:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5556 [2026-10-20 16:49:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9951 [2026-10-20 16:49:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8519 [2026-10-20 16:49:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9470 [2026-10-20 16:50:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8820 [2026-10-20 16:50:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4464 [2026-10-20 16:50:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3445 [2026-10-20 16:50:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0377 [2026-10-20 16:50:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5689 [2026-10-20 16:50:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0188 [2026-10-20 16:50:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7530 [2026-10-20 16:50:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7511 [2026-10-20 16:50:31][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,589 | MFU: 1.8% [2026-10-20 16:50:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4061 [2026-10-20 16:50:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9375 [2026-10-20 16:50:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7760 [2026-10-20 16:50:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7474 [2026-10-20 16:50:36][INFO][training.pretrain_engine] [Ph2|Step 555] loss=5.7224 | ppl=305.65 | aux=0.02509 | lr=3.00e-04 | gnorm=0.742 | tok/s=29,259 | MFU=7.0% | step=143350ms | tokens=0.589B | mem=4.2/32.7GB | cost=$291.66/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 09:39 (136.8h) | Phase3: SKIPPED | SFT: Oct 26 14:24 (141.6h) | DPO (Final): Oct 26 14:58 (142.1h) [2026-10-20 16:50:36][INFO][training.pretrain_engine] Starting global step 556 (accumulating 32 micro-batches)... [2026-10-20 16:50:36][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 16:50:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1326 [2026-10-20 16:50:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6890 [2026-10-20 16:50:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6452 [2026-10-20 16:50:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2202 [2026-10-20 16:51:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5550 [2026-10-20 16:51:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9557 [2026-10-20 16:51:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9728 [2026-10-20 16:51:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7818 [2026-10-20 16:51:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8455 [2026-10-20 16:51:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7631 [2026-10-20 16:51:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2895 [2026-10-20 16:51:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7317 [2026-10-20 16:51:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2851 [2026-10-20 16:51:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.5696 [2026-10-20 16:51:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1050 [2026-10-20 16:51:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1667 [2026-10-20 16:52:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5286 [2026-10-20 16:52:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3660 [2026-10-20 16:52:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7786 [2026-10-20 16:52:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4122 [2026-10-20 16:52:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8880 [2026-10-20 16:52:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6693 [2026-10-20 16:52:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8336 [2026-10-20 16:52:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0049 [2026-10-20 16:52:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1751 [2026-10-20 16:52:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2339 [2026-10-20 16:52:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1188 [2026-10-20 16:52:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1040 [2026-10-20 16:52:55][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,564 | MFU: 1.8% [2026-10-20 16:53:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5651 [2026-10-20 16:53:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7564 [2026-10-20 16:53:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6623 [2026-10-20 16:53:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6742 [2026-10-20 16:53:00][INFO][training.pretrain_engine] [Ph2|Step 556] loss=5.6868 | ppl=294.93 | aux=0.02509 | lr=3.00e-04 | gnorm=0.719 | tok/s=29,258 | MFU=7.0% | step=143354ms | tokens=0.590B | mem=4.2/32.7GB | cost=$292.19/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 09:41 (136.8h) | Phase3: SKIPPED | SFT: Oct 26 14:26 (141.6h) | DPO (Final): Oct 26 15:00 (142.1h) [2026-10-20 16:53:00][INFO][training.pretrain_engine] Starting global step 557 (accumulating 32 micro-batches)... [2026-10-20 16:53:00][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 16:53:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5945 [2026-10-20 16:53:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7648 [2026-10-20 16:53:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8401 [2026-10-20 16:53:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6494 [2026-10-20 16:53:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8991 [2026-10-20 16:53:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8453 [2026-10-20 16:53:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0303 [2026-10-20 16:53:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0218 [2026-10-20 16:53:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1465 [2026-10-20 16:53:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5960 [2026-10-20 16:53:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7974 [2026-10-20 16:53:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1355 [2026-10-20 16:54:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9478 [2026-10-20 16:54:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2058 [2026-10-20 16:54:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7963 [2026-10-20 16:54:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0117 [2026-10-20 16:54:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9090 [2026-10-20 16:54:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0615 [2026-10-20 16:54:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1048 [2026-10-20 16:54:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0647 [2026-10-20 16:54:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7969 [2026-10-20 16:54:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9898 [2026-10-20 16:54:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6157 [2026-10-20 16:54:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.9829 [2026-10-20 16:55:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2940 [2026-10-20 16:55:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6499 [2026-10-20 16:55:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4934 [2026-10-20 16:55:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5628 [2026-10-20 16:55:19][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,542 | MFU: 1.8% [2026-10-20 16:55:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6001 [2026-10-20 16:55:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0382 [2026-10-20 16:55:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7919 [2026-10-20 16:55:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0564 [2026-10-20 16:55:24][INFO][training.pretrain_engine] [Ph2|Step 557] loss=5.8273 | ppl=339.42 | aux=0.02510 | lr=3.00e-04 | gnorm=0.754 | tok/s=29,258 | MFU=7.0% | step=143358ms | tokens=0.591B | mem=4.2/32.7GB | cost=$292.71/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 09:43 (136.8h) | Phase3: SKIPPED | SFT: Oct 26 14:28 (141.5h) | DPO (Final): Oct 26 15:02 (142.1h) [2026-10-20 16:55:24][INFO][training.pretrain_engine] Starting global step 558 (accumulating 32 micro-batches)... [2026-10-20 16:55:24][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 16:55:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3027 [2026-10-20 16:55:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6698 [2026-10-20 16:55:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8378 [2026-10-20 16:55:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9432 [2026-10-20 16:55:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4101 [2026-10-20 16:55:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6249 [2026-10-20 16:55:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4471 [2026-10-20 16:55:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8264 [2026-10-20 16:56:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5857 [2026-10-20 16:56:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8546 [2026-10-20 16:56:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0063 [2026-10-20 16:56:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7375 [2026-10-20 16:56:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0231 [2026-10-20 16:56:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8864 [2026-10-20 16:56:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1523 [2026-10-20 16:56:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9703 [2026-10-20 16:56:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0722 [2026-10-20 16:56:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8982 [2026-10-20 16:56:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7119 [2026-10-20 16:56:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8238 [2026-10-20 16:57:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7777 [2026-10-20 16:57:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7680 [2026-10-20 16:57:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1465 [2026-10-20 16:57:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9037 [2026-10-20 16:57:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8642 [2026-10-20 16:57:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8258 [2026-10-20 16:57:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7326 [2026-10-20 16:57:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8488 [2026-10-20 16:57:42][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,573 | MFU: 1.8% [2026-10-20 16:57:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9250 [2026-10-20 16:57:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5482 [2026-10-20 16:57:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5721 [2026-10-20 16:57:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1263 [2026-10-20 16:57:47][INFO][training.pretrain_engine] [Ph2|Step 558] loss=5.7712 | ppl=320.94 | aux=0.02508 | lr=3.00e-04 | gnorm=0.727 | tok/s=29,258 | MFU=7.0% | step=143356ms | tokens=0.592B | mem=4.2/32.7GB | cost=$293.24/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 09:44 (136.8h) | Phase3: SKIPPED | SFT: Oct 26 14:29 (141.5h) | DPO (Final): Oct 26 15:03 (142.1h) [2026-10-20 16:57:47][INFO][training.pretrain_engine] Starting global step 559 (accumulating 32 micro-batches)... [2026-10-20 16:57:47][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 16:58:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5838 [2026-10-20 16:58:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5813 [2026-10-20 16:58:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7879 [2026-10-20 16:58:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5094 [2026-10-20 16:58:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1650 [2026-10-20 16:58:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8521 [2026-10-20 16:58:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0348 [2026-10-20 16:58:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0665 [2026-10-20 16:58:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9077 [2026-10-20 16:58:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8069 [2026-10-20 16:58:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9299 [2026-10-20 16:58:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7638 [2026-10-20 16:58:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7186 [2026-10-20 16:58:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2089 [2026-10-20 16:58:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1160 [2026-10-20 16:58:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3224 [2026-10-20 16:59:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3136 [2026-10-20 16:59:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6741 [2026-10-20 16:59:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5526 [2026-10-20 16:59:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9365 [2026-10-20 16:59:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4134 [2026-10-20 16:59:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5219 [2026-10-20 16:59:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7960 [2026-10-20 16:59:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7641 [2026-10-20 16:59:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2778 [2026-10-20 16:59:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0813 [2026-10-20 16:59:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1651 [2026-10-20 16:59:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0181 [2026-10-20 17:00:05][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,591 | MFU: 1.8% [2026-10-20 17:00:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8876 [2026-10-20 17:00:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8827 [2026-10-20 17:00:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4342 [2026-10-20 17:00:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2159 [2026-10-20 17:00:10][INFO][training.pretrain_engine] [Ph2|Step 559] loss=5.7065 | ppl=300.82 | aux=0.02509 | lr=3.00e-04 | gnorm=0.645 | tok/s=29,258 | MFU=7.0% | step=143353ms | tokens=0.593B | mem=4.2/32.7GB | cost=$293.76/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 09:46 (136.8h) | Phase3: SKIPPED | SFT: Oct 26 14:31 (141.5h) | DPO (Final): Oct 26 15:05 (142.1h) [2026-10-20 17:00:10][INFO][training.pretrain_engine] Starting global step 560 (accumulating 32 micro-batches)... [2026-10-20 17:00:10][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 17:00:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0158 [2026-10-20 17:00:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3925 [2026-10-20 17:00:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4537 [2026-10-20 17:00:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6410 [2026-10-20 17:00:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4159 [2026-10-20 17:00:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1924 [2026-10-20 17:00:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6260 [2026-10-20 17:00:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5311 [2026-10-20 17:01:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8480 [2026-10-20 17:01:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0454 [2026-10-20 17:01:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6194 [2026-10-20 17:01:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6952 [2026-10-20 17:01:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7184 [2026-10-20 17:01:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1069 [2026-10-20 17:01:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3894 [2026-10-20 17:01:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8077 [2026-10-20 17:01:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4903 [2026-10-20 17:01:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7876 [2026-10-20 17:01:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9237 [2026-10-20 17:01:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6760 [2026-10-20 17:01:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5193 [2026-10-20 17:01:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0591 [2026-10-20 17:01:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6899 [2026-10-20 17:01:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3717 [2026-10-20 17:02:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2688 [2026-10-20 17:02:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3946 [2026-10-20 17:02:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8096 [2026-10-20 17:02:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8080 [2026-10-20 17:02:28][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,590 | MFU: 1.8% [2026-10-20 17:02:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5733 [2026-10-20 17:02:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8513 [2026-10-20 17:02:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3817 [2026-10-20 17:02:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6383 [2026-10-20 17:02:33][INFO][training.pretrain_engine] [Ph2|Step 560] loss=5.7315 | ppl=308.44 | aux=0.02510 | lr=3.00e-04 | gnorm=0.676 | tok/s=29,259 | MFU=7.0% | step=143352ms | tokens=0.595B | mem=4.2/32.7GB | cost=$294.28/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 09:48 (136.8h) | Phase3: SKIPPED | SFT: Oct 26 14:33 (141.5h) | DPO (Final): Oct 26 15:07 (142.1h) [2026-10-20 17:02:36][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step560 | Tokens: 0.595B | Spent: $294.29 | Remaining: $105.71 [2026-10-20 17:02:36][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step560 [2026-10-20 17:02:36][INFO][training.pretrain_engine] Starting global step 561 (accumulating 32 micro-batches)... [2026-10-20 17:02:36][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 17:02:36][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 17:02:36][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 17:02:36][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 17:02:36][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step560 to Phase-Technologies/forge-500m-pretrain (path: phase2_step560)... [2026-10-20 17:02:36][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 17:02:46][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 17:02:46][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step560 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 17:02:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7244 [2026-10-20 17:02:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8187 [2026-10-20 17:02:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9456 [2026-10-20 17:02:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0634 [2026-10-20 17:03:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0267 [2026-10-20 17:03:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3483 [2026-10-20 17:03:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9840 [2026-10-20 17:03:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6938 [2026-10-20 17:03:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0901 [2026-10-20 17:03:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1225 [2026-10-20 17:03:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6749 [2026-10-20 17:03:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1533 [2026-10-20 17:03:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4871 [2026-10-20 17:03:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5702 [2026-10-20 17:03:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7634 [2026-10-20 17:03:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8730 [2026-10-20 17:04:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5590 [2026-10-20 17:04:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6934 [2026-10-20 17:04:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3240 [2026-10-20 17:04:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8515 [2026-10-20 17:04:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8792 [2026-10-20 17:04:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=3.9900 [2026-10-20 17:04:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8818 [2026-10-20 17:04:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2582 [2026-10-20 17:04:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7674 [2026-10-20 17:04:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1166 [2026-10-20 17:04:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2849 [2026-10-20 17:04:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8006 [2026-10-20 17:04:55][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,546 | MFU: 1.8% [2026-10-20 17:04:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6030 [2026-10-20 17:04:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2482 [2026-10-20 17:04:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8529 [2026-10-20 17:04:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9754 [2026-10-20 17:04:59][INFO][training.pretrain_engine] [Ph2|Step 561] loss=5.7741 | ppl=321.87 | aux=0.02508 | lr=3.00e-04 | gnorm=0.848 | tok/s=29,258 | MFU=7.0% | step=143356ms | tokens=0.596B | mem=4.2/32.7GB | cost=$294.82/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 09:50 (136.8h) | Phase3: SKIPPED | SFT: Oct 26 14:35 (141.5h) | DPO (Final): Oct 26 15:09 (142.1h) [2026-10-20 17:04:59][INFO][training.pretrain_engine] Starting global step 562 (accumulating 32 micro-batches)... [2026-10-20 17:04:59][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 17:05:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7355 [2026-10-20 17:05:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8341 [2026-10-20 17:05:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9747 [2026-10-20 17:05:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7005 [2026-10-20 17:05:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6411 [2026-10-20 17:05:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5685 [2026-10-20 17:05:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8338 [2026-10-20 17:05:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9179 [2026-10-20 17:05:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0412 [2026-10-20 17:05:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7542 [2026-10-20 17:05:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5773 [2026-10-20 17:05:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0700 [2026-10-20 17:06:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2683 [2026-10-20 17:06:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3452 [2026-10-20 17:06:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5635 [2026-10-20 17:06:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5874 [2026-10-20 17:06:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7049 [2026-10-20 17:06:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3581 [2026-10-20 17:06:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4565 [2026-10-20 17:06:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8006 [2026-10-20 17:06:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6794 [2026-10-20 17:06:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7524 [2026-10-20 17:06:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4100 [2026-10-20 17:06:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7227 [2026-10-20 17:07:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4619 [2026-10-20 17:07:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6609 [2026-10-20 17:07:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0890 [2026-10-20 17:07:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3564 [2026-10-20 17:07:17][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,583 | MFU: 1.8% [2026-10-20 17:07:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3547 [2026-10-20 17:07:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3174 [2026-10-20 17:07:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8708 [2026-10-20 17:07:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7263 [2026-10-20 17:07:22][INFO][training.pretrain_engine] [Ph2|Step 562] loss=5.5975 | ppl=269.75 | aux=0.02510 | lr=3.00e-04 | gnorm=1.047 | tok/s=29,259 | MFU=7.0% | step=143353ms | tokens=0.597B | mem=4.2/32.7GB | cost=$295.34/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 09:52 (136.7h) | Phase3: SKIPPED | SFT: Oct 26 14:36 (141.5h) | DPO (Final): Oct 26 15:11 (142.1h) [2026-10-20 17:07:22][INFO][training.pretrain_engine] Starting global step 563 (accumulating 32 micro-batches)... [2026-10-20 17:07:22][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 17:07:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4868 [2026-10-20 17:07:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2064 [2026-10-20 17:07:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8574 [2026-10-20 17:07:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8320 [2026-10-20 17:07:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0119 [2026-10-20 17:07:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4215 [2026-10-20 17:07:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5749 [2026-10-20 17:07:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5821 [2026-10-20 17:08:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0427 [2026-10-20 17:08:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3925 [2026-10-20 17:08:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9045 [2026-10-20 17:08:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7755 [2026-10-20 17:08:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1050 [2026-10-20 17:08:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2623 [2026-10-20 17:08:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8740 [2026-10-20 17:08:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1190 [2026-10-20 17:08:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8198 [2026-10-20 17:08:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3466 [2026-10-20 17:08:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6228 [2026-10-20 17:08:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1844 [2026-10-20 17:09:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1646 [2026-10-20 17:09:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6154 [2026-10-20 17:09:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9408 [2026-10-20 17:09:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8948 [2026-10-20 17:09:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6739 [2026-10-20 17:09:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1461 [2026-10-20 17:09:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1791 [2026-10-20 17:09:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1396 [2026-10-20 17:09:40][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,608 | MFU: 1.8% [2026-10-20 17:09:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0112 [2026-10-20 17:09:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5477 [2026-10-20 17:09:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2329 [2026-10-20 17:09:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7114 [2026-10-20 17:09:45][INFO][training.pretrain_engine] [Ph2|Step 563] loss=5.7797 | ppl=323.68 | aux=0.02509 | lr=3.00e-04 | gnorm=1.305 | tok/s=29,259 | MFU=7.0% | step=143350ms | tokens=0.598B | mem=4.2/32.7GB | cost=$295.86/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 09:53 (136.7h) | Phase3: SKIPPED | SFT: Oct 26 14:38 (141.5h) | DPO (Final): Oct 26 15:12 (142.0h) [2026-10-20 17:09:45][INFO][training.pretrain_engine] Starting global step 564 (accumulating 32 micro-batches)... [2026-10-20 17:09:45][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 17:10:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8128 [2026-10-20 17:10:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9497 [2026-10-20 17:10:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7590 [2026-10-20 17:10:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6014 [2026-10-20 17:10:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6911 [2026-10-20 17:10:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7083 [2026-10-20 17:10:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6013 [2026-10-20 17:10:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7250 [2026-10-20 17:10:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5511 [2026-10-20 17:10:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2189 [2026-10-20 17:10:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0360 [2026-10-20 17:10:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7465 [2026-10-20 17:10:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2130 [2026-10-20 17:10:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3034 [2026-10-20 17:10:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4509 [2026-10-20 17:10:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6558 [2026-10-20 17:11:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6914 [2026-10-20 17:11:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3667 [2026-10-20 17:11:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3189 [2026-10-20 17:11:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7343 [2026-10-20 17:11:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0079 [2026-10-20 17:11:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2232 [2026-10-20 17:11:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0594 [2026-10-20 17:11:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2101 [2026-10-20 17:11:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9932 [2026-10-20 17:11:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6466 [2026-10-20 17:11:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8903 [2026-10-20 17:11:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9952 [2026-10-20 17:12:04][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,549 | MFU: 1.8% [2026-10-20 17:12:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3657 [2026-10-20 17:12:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0941 [2026-10-20 17:12:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2774 [2026-10-20 17:12:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5224 [2026-10-20 17:12:09][INFO][training.pretrain_engine] [Ph2|Step 564] loss=5.6039 | ppl=271.49 | aux=0.02508 | lr=3.00e-04 | gnorm=0.848 | tok/s=29,259 | MFU=7.0% | step=143352ms | tokens=0.599B | mem=4.2/32.7GB | cost=$296.39/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 09:55 (136.7h) | Phase3: SKIPPED | SFT: Oct 26 14:40 (141.5h) | DPO (Final): Oct 26 15:14 (142.0h) [2026-10-20 17:12:09][INFO][training.pretrain_engine] Starting global step 565 (accumulating 32 micro-batches)... [2026-10-20 17:12:09][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 17:12:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8581 [2026-10-20 17:12:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8299 [2026-10-20 17:12:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0976 [2026-10-20 17:12:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6133 [2026-10-20 17:12:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7414 [2026-10-20 17:12:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8874 [2026-10-20 17:12:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9519 [2026-10-20 17:12:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8737 [2026-10-20 17:13:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5234 [2026-10-20 17:13:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3677 [2026-10-20 17:13:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6223 [2026-10-20 17:13:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8595 [2026-10-20 17:13:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0658 [2026-10-20 17:13:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4245 [2026-10-20 17:13:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9949 [2026-10-20 17:13:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8841 [2026-10-20 17:13:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6157 [2026-10-20 17:13:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8719 [2026-10-20 17:13:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4037 [2026-10-20 17:13:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5953 [2026-10-20 17:13:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8100 [2026-10-20 17:13:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8019 [2026-10-20 17:13:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2899 [2026-10-20 17:13:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0103 [2026-10-20 17:14:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1277 [2026-10-20 17:14:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1143 [2026-10-20 17:14:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2653 [2026-10-20 17:14:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6481 [2026-10-20 17:14:27][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,591 | MFU: 1.8% [2026-10-20 17:14:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3991 [2026-10-20 17:14:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9028 [2026-10-20 17:14:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1051 [2026-10-20 17:14:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7521 [2026-10-20 17:14:32][INFO][training.pretrain_engine] [Ph2|Step 565] loss=5.8137 | ppl=334.86 | aux=0.02508 | lr=3.00e-04 | gnorm=1.344 | tok/s=29,259 | MFU=7.0% | step=143353ms | tokens=0.600B | mem=4.2/32.7GB | cost=$296.91/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 09:57 (136.7h) | Phase3: SKIPPED | SFT: Oct 26 14:42 (141.5h) | DPO (Final): Oct 26 15:16 (142.0h) [2026-10-20 17:14:32][INFO][training.pretrain_engine] Starting global step 566 (accumulating 32 micro-batches)... [2026-10-20 17:14:32][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 17:14:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7616 [2026-10-20 17:14:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6415 [2026-10-20 17:14:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8180 [2026-10-20 17:14:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4764 [2026-10-20 17:15:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5195 [2026-10-20 17:15:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5987 [2026-10-20 17:15:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4014 [2026-10-20 17:15:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0075 [2026-10-20 17:15:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0287 [2026-10-20 17:15:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8902 [2026-10-20 17:15:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7707 [2026-10-20 17:15:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9458 [2026-10-20 17:15:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8596 [2026-10-20 17:15:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3864 [2026-10-20 17:15:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6807 [2026-10-20 17:15:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5268 [2026-10-20 17:16:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5504 [2026-10-20 17:16:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6702 [2026-10-20 17:16:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4869 [2026-10-20 17:16:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8095 [2026-10-20 17:16:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1641 [2026-10-20 17:16:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8518 [2026-10-20 17:16:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3110 [2026-10-20 17:16:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7486 [2026-10-20 17:16:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4777 [2026-10-20 17:16:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7526 [2026-10-20 17:16:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7225 [2026-10-20 17:16:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8696 [2026-10-20 17:16:50][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,605 | MFU: 1.8% [2026-10-20 17:16:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8961 [2026-10-20 17:16:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8598 [2026-10-20 17:16:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9886 [2026-10-20 17:16:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5892 [2026-10-20 17:16:55][INFO][training.pretrain_engine] [Ph2|Step 566] loss=5.7487 | ppl=313.79 | aux=0.02507 | lr=3.00e-04 | gnorm=0.969 | tok/s=29,259 | MFU=7.0% | step=143351ms | tokens=0.601B | mem=4.2/32.7GB | cost=$297.44/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 09:59 (136.7h) | Phase3: SKIPPED | SFT: Oct 26 14:43 (141.4h) | DPO (Final): Oct 26 15:18 (142.0h) [2026-10-20 17:16:55][INFO][training.pretrain_engine] Starting global step 567 (accumulating 32 micro-batches)... [2026-10-20 17:16:55][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 17:17:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7204 [2026-10-20 17:17:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7970 [2026-10-20 17:17:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9305 [2026-10-20 17:17:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7954 [2026-10-20 17:17:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2200 [2026-10-20 17:17:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2560 [2026-10-20 17:17:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7828 [2026-10-20 17:17:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3416 [2026-10-20 17:17:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8290 [2026-10-20 17:17:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4746 [2026-10-20 17:17:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1028 [2026-10-20 17:17:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8668 [2026-10-20 17:18:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6875 [2026-10-20 17:18:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6487 [2026-10-20 17:18:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5988 [2026-10-20 17:18:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5446 [2026-10-20 17:18:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6999 [2026-10-20 17:18:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3378 [2026-10-20 17:18:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8237 [2026-10-20 17:18:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5285 [2026-10-20 17:18:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8429 [2026-10-20 17:18:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7098 [2026-10-20 17:18:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6636 [2026-10-20 17:18:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7878 [2026-10-20 17:19:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3057 [2026-10-20 17:19:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0790 [2026-10-20 17:19:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4233 [2026-10-20 17:19:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2475 [2026-10-20 17:19:13][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,600 | MFU: 1.8% [2026-10-20 17:19:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8234 [2026-10-20 17:19:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2827 [2026-10-20 17:19:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8461 [2026-10-20 17:19:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6750 [2026-10-20 17:19:18][INFO][training.pretrain_engine] [Ph2|Step 567] loss=5.7099 | ppl=301.84 | aux=0.02508 | lr=3.00e-04 | gnorm=1.031 | tok/s=29,259 | MFU=7.0% | step=143353ms | tokens=0.602B | mem=4.2/32.7GB | cost=$297.96/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 10:00 (136.7h) | Phase3: SKIPPED | SFT: Oct 26 14:45 (141.4h) | DPO (Final): Oct 26 15:19 (142.0h) [2026-10-20 17:19:18][INFO][training.pretrain_engine] Starting global step 568 (accumulating 32 micro-batches)... [2026-10-20 17:19:18][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 17:19:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6589 [2026-10-20 17:19:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7058 [2026-10-20 17:19:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5082 [2026-10-20 17:19:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4926 [2026-10-20 17:19:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0363 [2026-10-20 17:19:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0586 [2026-10-20 17:19:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3749 [2026-10-20 17:19:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0092 [2026-10-20 17:20:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3501 [2026-10-20 17:20:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1432 [2026-10-20 17:20:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9506 [2026-10-20 17:20:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5143 [2026-10-20 17:20:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3067 [2026-10-20 17:20:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4599 [2026-10-20 17:20:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0358 [2026-10-20 17:20:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6702 [2026-10-20 17:20:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3105 [2026-10-20 17:20:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7206 [2026-10-20 17:20:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5570 [2026-10-20 17:20:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5251 [2026-10-20 17:21:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6277 [2026-10-20 17:21:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5686 [2026-10-20 17:21:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6000 [2026-10-20 17:21:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4932 [2026-10-20 17:21:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4661 [2026-10-20 17:21:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3290 [2026-10-20 17:21:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5790 [2026-10-20 17:21:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3552 [2026-10-20 17:21:37][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,559 | MFU: 1.8% [2026-10-20 17:21:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9446 [2026-10-20 17:21:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7963 [2026-10-20 17:21:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0580 [2026-10-20 17:21:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8137 [2026-10-20 17:21:42][INFO][training.pretrain_engine] [Ph2|Step 568] loss=5.6252 | ppl=277.33 | aux=0.02509 | lr=3.00e-04 | gnorm=1.109 | tok/s=29,257 | MFU=7.0% | step=143359ms | tokens=0.603B | mem=4.2/32.7GB | cost=$298.48/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 10:03 (136.7h) | Phase3: SKIPPED | SFT: Oct 26 14:47 (141.4h) | DPO (Final): Oct 26 15:22 (142.0h) [2026-10-20 17:21:42][INFO][training.pretrain_engine] Starting global step 569 (accumulating 32 micro-batches)... [2026-10-20 17:21:42][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 17:21:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5814 [2026-10-20 17:21:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5769 [2026-10-20 17:21:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3080 [2026-10-20 17:22:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5241 [2026-10-20 17:22:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7913 [2026-10-20 17:22:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7916 [2026-10-20 17:22:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6511 [2026-10-20 17:22:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9444 [2026-10-20 17:22:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7706 [2026-10-20 17:22:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8897 [2026-10-20 17:22:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2084 [2026-10-20 17:22:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7550 [2026-10-20 17:22:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7639 [2026-10-20 17:22:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9850 [2026-10-20 17:22:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6321 [2026-10-20 17:22:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8771 [2026-10-20 17:23:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2725 [2026-10-20 17:23:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3335 [2026-10-20 17:23:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4308 [2026-10-20 17:23:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3807 [2026-10-20 17:23:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4193 [2026-10-20 17:23:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2514 [2026-10-20 17:23:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6436 [2026-10-20 17:23:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5037 [2026-10-20 17:23:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8624 [2026-10-20 17:23:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3169 [2026-10-20 17:23:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2702 [2026-10-20 17:23:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7686 [2026-10-20 17:24:00][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,569 | MFU: 1.8% [2026-10-20 17:24:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9220 [2026-10-20 17:24:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2029 [2026-10-20 17:24:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6196 [2026-10-20 17:24:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9517 [2026-10-20 17:24:05][INFO][training.pretrain_engine] [Ph2|Step 569] loss=5.7486 | ppl=313.75 | aux=0.02511 | lr=3.00e-04 | gnorm=0.941 | tok/s=29,257 | MFU=7.0% | step=143363ms | tokens=0.604B | mem=4.2/32.7GB | cost=$299.01/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 10:05 (136.7h) | Phase3: SKIPPED | SFT: Oct 26 14:49 (141.4h) | DPO (Final): Oct 26 15:24 (142.0h) [2026-10-20 17:24:08][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step569 | Tokens: 0.604B | Spent: $299.02 | Remaining: $100.98 [2026-10-20 17:24:08][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step569 [2026-10-20 17:24:08][INFO][training.pretrain_engine] Starting global step 570 (accumulating 32 micro-batches)... [2026-10-20 17:24:08][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 17:24:09][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 17:24:09][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 17:24:09][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 17:24:09][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step569 to Phase-Technologies/forge-500m-pretrain (path: phase2_step569)... [2026-10-20 17:24:09][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 17:24:18][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 17:24:18][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step569 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 17:24:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5852 [2026-10-20 17:24:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0759 [2026-10-20 17:24:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6867 [2026-10-20 17:24:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5061 [2026-10-20 17:24:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0943 [2026-10-20 17:24:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8037 [2026-10-20 17:24:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7756 [2026-10-20 17:24:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1261 [2026-10-20 17:24:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7530 [2026-10-20 17:24:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8052 [2026-10-20 17:24:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7583 [2026-10-20 17:25:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7271 [2026-10-20 17:25:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5937 [2026-10-20 17:25:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3446 [2026-10-20 17:25:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.1700 [2026-10-20 17:25:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1021 [2026-10-20 17:25:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8124 [2026-10-20 17:25:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9680 [2026-10-20 17:25:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7228 [2026-10-20 17:25:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8084 [2026-10-20 17:25:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5267 [2026-10-20 17:25:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6217 [2026-10-20 17:25:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5655 [2026-10-20 17:25:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5304 [2026-10-20 17:26:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8109 [2026-10-20 17:26:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5032 [2026-10-20 17:26:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2175 [2026-10-20 17:26:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.3108 [2026-10-20 17:26:27][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,569 | MFU: 1.8% [2026-10-20 17:26:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0306 [2026-10-20 17:26:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1390 [2026-10-20 17:26:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7847 [2026-10-20 17:26:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1824 [2026-10-20 17:26:31][INFO][training.pretrain_engine] [Ph2|Step 570] loss=5.7571 | ppl=316.42 | aux=0.02508 | lr=3.00e-04 | gnorm=0.812 | tok/s=29,257 | MFU=7.0% | step=143362ms | tokens=0.605B | mem=4.2/32.7GB | cost=$299.54/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 10:06 (136.7h) | Phase3: SKIPPED | SFT: Oct 26 14:51 (141.4h) | DPO (Final): Oct 26 15:25 (142.0h) [2026-10-20 17:26:31][INFO][training.pretrain_engine] Starting global step 571 (accumulating 32 micro-batches)... [2026-10-20 17:26:31][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 17:26:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0224 [2026-10-20 17:26:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7740 [2026-10-20 17:26:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1087 [2026-10-20 17:26:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2983 [2026-10-20 17:27:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6155 [2026-10-20 17:27:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3161 [2026-10-20 17:27:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1164 [2026-10-20 17:27:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1855 [2026-10-20 17:27:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6806 [2026-10-20 17:27:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6752 [2026-10-20 17:27:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5443 [2026-10-20 17:27:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2149 [2026-10-20 17:27:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7361 [2026-10-20 17:27:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7003 [2026-10-20 17:27:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9634 [2026-10-20 17:27:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5266 [2026-10-20 17:28:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6973 [2026-10-20 17:28:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7228 [2026-10-20 17:28:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8512 [2026-10-20 17:28:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5330 [2026-10-20 17:28:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1873 [2026-10-20 17:28:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2121 [2026-10-20 17:28:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1854 [2026-10-20 17:28:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5809 [2026-10-20 17:28:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8870 [2026-10-20 17:28:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8609 [2026-10-20 17:28:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8152 [2026-10-20 17:28:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3497 [2026-10-20 17:28:50][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,562 | MFU: 1.8% [2026-10-20 17:28:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8526 [2026-10-20 17:28:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0723 [2026-10-20 17:28:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8134 [2026-10-20 17:28:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4158 [2026-10-20 17:28:55][INFO][training.pretrain_engine] [Ph2|Step 571] loss=5.7766 | ppl=322.65 | aux=0.02511 | lr=3.00e-04 | gnorm=0.859 | tok/s=29,257 | MFU=7.0% | step=143362ms | tokens=0.606B | mem=4.2/32.7GB | cost=$300.07/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 10:08 (136.7h) | Phase3: SKIPPED | SFT: Oct 26 14:53 (141.4h) | DPO (Final): Oct 26 15:27 (142.0h) [2026-10-20 17:28:55][INFO][training.pretrain_engine] Starting global step 572 (accumulating 32 micro-batches)... [2026-10-20 17:28:55][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 17:29:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9561 [2026-10-20 17:29:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3148 [2026-10-20 17:29:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7302 [2026-10-20 17:29:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4317 [2026-10-20 17:29:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3427 [2026-10-20 17:29:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2624 [2026-10-20 17:29:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1617 [2026-10-20 17:29:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5477 [2026-10-20 17:29:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4308 [2026-10-20 17:29:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2923 [2026-10-20 17:29:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7486 [2026-10-20 17:29:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4692 [2026-10-20 17:30:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4723 [2026-10-20 17:30:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6058 [2026-10-20 17:30:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6232 [2026-10-20 17:30:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6402 [2026-10-20 17:30:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8303 [2026-10-20 17:30:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5997 [2026-10-20 17:30:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8142 [2026-10-20 17:30:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8806 [2026-10-20 17:30:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0552 [2026-10-20 17:30:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8150 [2026-10-20 17:30:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9140 [2026-10-20 17:30:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5180 [2026-10-20 17:30:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2647 [2026-10-20 17:31:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8218 [2026-10-20 17:31:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2423 [2026-10-20 17:31:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=7.5062 [2026-10-20 17:31:13][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,567 | MFU: 1.8% [2026-10-20 17:31:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9562 [2026-10-20 17:31:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1605 [2026-10-20 17:31:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4761 [2026-10-20 17:31:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1789 [2026-10-20 17:31:18][INFO][training.pretrain_engine] [Ph2|Step 572] loss=5.6340 | ppl=279.77 | aux=0.02510 | lr=3.00e-04 | gnorm=0.777 | tok/s=29,257 | MFU=7.0% | step=143361ms | tokens=0.607B | mem=4.2/32.7GB | cost=$300.59/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 10:10 (136.7h) | Phase3: SKIPPED | SFT: Oct 26 14:55 (141.4h) | DPO (Final): Oct 26 15:29 (142.0h) [2026-10-20 17:31:18][INFO][training.pretrain_engine] Starting global step 573 (accumulating 32 micro-batches)... [2026-10-20 17:31:18][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 17:31:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8531 [2026-10-20 17:31:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3896 [2026-10-20 17:31:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5146 [2026-10-20 17:31:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9064 [2026-10-20 17:31:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8014 [2026-10-20 17:31:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7944 [2026-10-20 17:31:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4786 [2026-10-20 17:31:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1143 [2026-10-20 17:32:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1505 [2026-10-20 17:32:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4560 [2026-10-20 17:32:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1290 [2026-10-20 17:32:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5428 [2026-10-20 17:32:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8475 [2026-10-20 17:32:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5423 [2026-10-20 17:32:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8559 [2026-10-20 17:32:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6628 [2026-10-20 17:32:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6398 [2026-10-20 17:32:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3611 [2026-10-20 17:32:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5040 [2026-10-20 17:32:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3409 [2026-10-20 17:33:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8992 [2026-10-20 17:33:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1698 [2026-10-20 17:33:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8240 [2026-10-20 17:33:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9021 [2026-10-20 17:33:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6748 [2026-10-20 17:33:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6716 [2026-10-20 17:33:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3249 [2026-10-20 17:33:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7792 [2026-10-20 17:33:37][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,562 | MFU: 1.8% [2026-10-20 17:33:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2978 [2026-10-20 17:33:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1067 [2026-10-20 17:33:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5971 [2026-10-20 17:33:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3620 [2026-10-20 17:33:41][INFO][training.pretrain_engine] [Ph2|Step 573] loss=5.7381 | ppl=310.46 | aux=0.02510 | lr=3.00e-04 | gnorm=0.742 | tok/s=29,257 | MFU=7.0% | step=143361ms | tokens=0.608B | mem=4.2/32.7GB | cost=$301.11/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 10:12 (136.6h) | Phase3: SKIPPED | SFT: Oct 26 14:57 (141.4h) | DPO (Final): Oct 26 15:31 (142.0h) [2026-10-20 17:33:41][INFO][training.pretrain_engine] Starting global step 574 (accumulating 32 micro-batches)... [2026-10-20 17:33:41][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 17:33:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8588 [2026-10-20 17:33:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1422 [2026-10-20 17:33:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7941 [2026-10-20 17:33:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4116 [2026-10-20 17:34:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2847 [2026-10-20 17:34:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6602 [2026-10-20 17:34:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0588 [2026-10-20 17:34:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4606 [2026-10-20 17:34:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2735 [2026-10-20 17:34:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0388 [2026-10-20 17:34:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7120 [2026-10-20 17:34:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6514 [2026-10-20 17:34:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6551 [2026-10-20 17:34:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8251 [2026-10-20 17:34:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4076 [2026-10-20 17:34:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9059 [2026-10-20 17:35:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1312 [2026-10-20 17:35:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1378 [2026-10-20 17:35:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9333 [2026-10-20 17:35:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2219 [2026-10-20 17:35:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0606 [2026-10-20 17:35:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9293 [2026-10-20 17:35:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8322 [2026-10-20 17:35:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9941 [2026-10-20 17:35:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5900 [2026-10-20 17:35:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7017 [2026-10-20 17:35:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7598 [2026-10-20 17:35:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8812 [2026-10-20 17:36:00][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,575 | MFU: 1.8% [2026-10-20 17:36:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5158 [2026-10-20 17:36:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8598 [2026-10-20 17:36:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7534 [2026-10-20 17:36:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9521 [2026-10-20 17:36:04][INFO][training.pretrain_engine] [Ph2|Step 574] loss=5.7500 | ppl=314.18 | aux=0.02509 | lr=3.00e-04 | gnorm=0.746 | tok/s=29,258 | MFU=7.0% | step=143357ms | tokens=0.609B | mem=4.2/32.7GB | cost=$301.64/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 10:13 (136.6h) | Phase3: SKIPPED | SFT: Oct 26 14:58 (141.4h) | DPO (Final): Oct 26 15:32 (141.9h) [2026-10-20 17:36:04][INFO][training.pretrain_engine] Starting global step 575 (accumulating 32 micro-batches)... [2026-10-20 17:36:04][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 17:36:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9719 [2026-10-20 17:36:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1403 [2026-10-20 17:36:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1760 [2026-10-20 17:36:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6497 [2026-10-20 17:36:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4252 [2026-10-20 17:36:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7631 [2026-10-20 17:36:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7418 [2026-10-20 17:36:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6477 [2026-10-20 17:36:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6544 [2026-10-20 17:36:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7265 [2026-10-20 17:36:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8868 [2026-10-20 17:36:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8785 [2026-10-20 17:37:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5662 [2026-10-20 17:37:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2509 [2026-10-20 17:37:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1159 [2026-10-20 17:37:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6396 [2026-10-20 17:37:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7413 [2026-10-20 17:37:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5288 [2026-10-20 17:37:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7728 [2026-10-20 17:37:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7448 [2026-10-20 17:37:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8047 [2026-10-20 17:37:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5752 [2026-10-20 17:37:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4722 [2026-10-20 17:37:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7449 [2026-10-20 17:38:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8804 [2026-10-20 17:38:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2247 [2026-10-20 17:38:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8371 [2026-10-20 17:38:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8670 [2026-10-20 17:38:23][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,566 | MFU: 1.8% [2026-10-20 17:38:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6984 [2026-10-20 17:38:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5525 [2026-10-20 17:38:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5472 [2026-10-20 17:38:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8666 [2026-10-20 17:38:28][INFO][training.pretrain_engine] [Ph2|Step 575] loss=5.5722 | ppl=263.02 | aux=0.02510 | lr=3.00e-04 | gnorm=0.711 | tok/s=29,257 | MFU=7.0% | step=143359ms | tokens=0.610B | mem=4.2/32.7GB | cost=$302.16/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 10:15 (136.6h) | Phase3: SKIPPED | SFT: Oct 26 15:00 (141.4h) | DPO (Final): Oct 26 15:34 (141.9h) [2026-10-20 17:38:28][INFO][training.pretrain_engine] Starting global step 576 (accumulating 32 micro-batches)... [2026-10-20 17:38:28][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 17:38:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8168 [2026-10-20 17:38:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8330 [2026-10-20 17:38:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8031 [2026-10-20 17:38:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8707 [2026-10-20 17:39:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9441 [2026-10-20 17:39:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3350 [2026-10-20 17:39:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8584 [2026-10-20 17:39:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9032 [2026-10-20 17:39:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9668 [2026-10-20 17:39:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0946 [2026-10-20 17:39:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5788 [2026-10-20 17:39:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2238 [2026-10-20 17:39:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7470 [2026-10-20 17:39:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1231 [2026-10-20 17:39:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2366 [2026-10-20 17:39:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7109 [2026-10-20 17:39:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6314 [2026-10-20 17:39:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1711 [2026-10-20 17:39:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8855 [2026-10-20 17:39:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6356 [2026-10-20 17:40:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.7848 [2026-10-20 17:40:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9684 [2026-10-20 17:40:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1221 [2026-10-20 17:40:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0482 [2026-10-20 17:40:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5815 [2026-10-20 17:40:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2431 [2026-10-20 17:40:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0201 [2026-10-20 17:40:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9920 [2026-10-20 17:40:46][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,562 | MFU: 1.8% [2026-10-20 17:40:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6817 [2026-10-20 17:40:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7713 [2026-10-20 17:40:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9231 [2026-10-20 17:40:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6440 [2026-10-20 17:40:51][INFO][training.pretrain_engine] [Ph2|Step 576] loss=5.7976 | ppl=329.50 | aux=0.02509 | lr=3.00e-04 | gnorm=0.645 | tok/s=29,258 | MFU=7.0% | step=143358ms | tokens=0.611B | mem=4.2/32.7GB | cost=$302.69/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 10:17 (136.6h) | Phase3: SKIPPED | SFT: Oct 26 15:02 (141.4h) | DPO (Final): Oct 26 15:36 (141.9h) [2026-10-20 17:40:51][INFO][training.pretrain_engine] Starting global step 577 (accumulating 32 micro-batches)... [2026-10-20 17:40:51][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 17:41:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4522 [2026-10-20 17:41:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7035 [2026-10-20 17:41:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5820 [2026-10-20 17:41:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4225 [2026-10-20 17:41:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7389 [2026-10-20 17:41:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0316 [2026-10-20 17:41:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4832 [2026-10-20 17:41:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9852 [2026-10-20 17:41:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7145 [2026-10-20 17:41:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7805 [2026-10-20 17:41:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7813 [2026-10-20 17:41:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1013 [2026-10-20 17:42:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4805 [2026-10-20 17:42:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8630 [2026-10-20 17:42:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6031 [2026-10-20 17:42:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3822 [2026-10-20 17:42:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7493 [2026-10-20 17:42:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5008 [2026-10-20 17:42:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0751 [2026-10-20 17:42:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5482 [2026-10-20 17:42:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8314 [2026-10-20 17:42:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1960 [2026-10-20 17:42:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0948 [2026-10-20 17:42:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5021 [2026-10-20 17:42:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0000 [2026-10-20 17:42:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9588 [2026-10-20 17:42:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5418 [2026-10-20 17:42:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5748 [2026-10-20 17:43:09][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,576 | MFU: 1.8% [2026-10-20 17:43:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0797 [2026-10-20 17:43:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6138 [2026-10-20 17:43:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9306 [2026-10-20 17:43:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2017 [2026-10-20 17:43:14][INFO][training.pretrain_engine] [Ph2|Step 577] loss=5.7238 | ppl=306.05 | aux=0.02507 | lr=3.00e-04 | gnorm=0.539 | tok/s=29,258 | MFU=7.0% | step=143358ms | tokens=0.612B | mem=4.2/32.7GB | cost=$303.21/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 10:19 (136.6h) | Phase3: SKIPPED | SFT: Oct 26 15:04 (141.3h) | DPO (Final): Oct 26 15:38 (141.9h) [2026-10-20 17:43:14][INFO][training.pretrain_engine] Starting global step 578 (accumulating 32 micro-batches)... [2026-10-20 17:43:14][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 17:43:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1304 [2026-10-20 17:43:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1473 [2026-10-20 17:43:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2835 [2026-10-20 17:43:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2107 [2026-10-20 17:43:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0184 [2026-10-20 17:43:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5557 [2026-10-20 17:43:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2640 [2026-10-20 17:43:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4740 [2026-10-20 17:44:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3353 [2026-10-20 17:44:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9350 [2026-10-20 17:44:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6377 [2026-10-20 17:44:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5803 [2026-10-20 17:44:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8036 [2026-10-20 17:44:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9609 [2026-10-20 17:44:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8614 [2026-10-20 17:44:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8082 [2026-10-20 17:44:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5628 [2026-10-20 17:44:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9181 [2026-10-20 17:44:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6809 [2026-10-20 17:44:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6949 [2026-10-20 17:45:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6543 [2026-10-20 17:45:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7389 [2026-10-20 17:45:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6449 [2026-10-20 17:45:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6309 [2026-10-20 17:45:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4893 [2026-10-20 17:45:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8205 [2026-10-20 17:45:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1330 [2026-10-20 17:45:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9488 [2026-10-20 17:45:33][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,567 | MFU: 1.8% [2026-10-20 17:45:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5840 [2026-10-20 17:45:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9382 [2026-10-20 17:45:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1735 [2026-10-20 17:45:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9649 [2026-10-20 17:45:37][INFO][training.pretrain_engine] [Ph2|Step 578] loss=5.7080 | ppl=301.28 | aux=0.02508 | lr=3.00e-04 | gnorm=0.547 | tok/s=29,257 | MFU=7.0% | step=143360ms | tokens=0.613B | mem=4.2/32.7GB | cost=$303.73/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 10:21 (136.6h) | Phase3: SKIPPED | SFT: Oct 26 15:05 (141.3h) | DPO (Final): Oct 26 15:40 (141.9h) [2026-10-20 17:45:41][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step578 | Tokens: 0.613B | Spent: $303.74 | Remaining: $96.26 [2026-10-20 17:45:41][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step578 [2026-10-20 17:45:41][INFO][training.pretrain_engine] Starting global step 579 (accumulating 32 micro-batches)... [2026-10-20 17:45:41][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 17:45:41][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 17:45:41][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 17:45:41][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 17:45:41][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step578 to Phase-Technologies/forge-500m-pretrain (path: phase2_step578)... [2026-10-20 17:45:41][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 17:45:51][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 17:45:51][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step578 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 17:45:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7517 [2026-10-20 17:45:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6082 [2026-10-20 17:45:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5713 [2026-10-20 17:45:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7594 [2026-10-20 17:46:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4223 [2026-10-20 17:46:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3816 [2026-10-20 17:46:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8071 [2026-10-20 17:46:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0594 [2026-10-20 17:46:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7914 [2026-10-20 17:46:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7023 [2026-10-20 17:46:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4278 [2026-10-20 17:46:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3368 [2026-10-20 17:46:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1843 [2026-10-20 17:46:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6563 [2026-10-20 17:46:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9258 [2026-10-20 17:46:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6640 [2026-10-20 17:47:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6373 [2026-10-20 17:47:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4142 [2026-10-20 17:47:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2134 [2026-10-20 17:47:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8010 [2026-10-20 17:47:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2709 [2026-10-20 17:47:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6054 [2026-10-20 17:47:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7798 [2026-10-20 17:47:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1529 [2026-10-20 17:47:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8154 [2026-10-20 17:47:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0755 [2026-10-20 17:47:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8510 [2026-10-20 17:47:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2181 [2026-10-20 17:48:00][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,512 | MFU: 1.8% [2026-10-20 17:48:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9594 [2026-10-20 17:48:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4588 [2026-10-20 17:48:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7281 [2026-10-20 17:48:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8874 [2026-10-20 17:48:05][INFO][training.pretrain_engine] [Ph2|Step 579] loss=5.8058 | ppl=332.22 | aux=0.02510 | lr=3.00e-04 | gnorm=0.559 | tok/s=29,256 | MFU=7.0% | step=143368ms | tokens=0.614B | mem=4.2/32.7GB | cost=$304.27/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 10:23 (136.6h) | Phase3: SKIPPED | SFT: Oct 26 15:08 (141.3h) | DPO (Final): Oct 26 15:42 (141.9h) [2026-10-20 17:48:05][INFO][training.pretrain_engine] Starting global step 580 (accumulating 32 micro-batches)... [2026-10-20 17:48:05][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 17:48:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9324 [2026-10-20 17:48:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1097 [2026-10-20 17:48:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2180 [2026-10-20 17:48:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7237 [2026-10-20 17:48:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8637 [2026-10-20 17:48:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2426 [2026-10-20 17:48:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7268 [2026-10-20 17:48:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5904 [2026-10-20 17:48:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6194 [2026-10-20 17:48:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6253 [2026-10-20 17:48:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3549 [2026-10-20 17:48:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4673 [2026-10-20 17:49:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1976 [2026-10-20 17:49:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8896 [2026-10-20 17:49:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2537 [2026-10-20 17:49:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9594 [2026-10-20 17:49:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7479 [2026-10-20 17:49:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0198 [2026-10-20 17:49:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5909 [2026-10-20 17:49:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7595 [2026-10-20 17:49:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4359 [2026-10-20 17:49:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7395 [2026-10-20 17:49:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6957 [2026-10-20 17:49:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3386 [2026-10-20 17:50:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0065 [2026-10-20 17:50:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9676 [2026-10-20 17:50:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5667 [2026-10-20 17:50:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7079 [2026-10-20 17:50:24][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,517 | MFU: 1.8% [2026-10-20 17:50:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8251 [2026-10-20 17:50:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2973 [2026-10-20 17:50:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8335 [2026-10-20 17:50:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0190 [2026-10-20 17:50:29][INFO][training.pretrain_engine] [Ph2|Step 580] loss=5.6467 | ppl=283.35 | aux=0.02508 | lr=3.00e-04 | gnorm=0.586 | tok/s=29,254 | MFU=7.0% | step=143376ms | tokens=0.616B | mem=4.2/32.7GB | cost=$304.80/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 10:25 (136.6h) | Phase3: SKIPPED | SFT: Oct 26 15:10 (141.3h) | DPO (Final): Oct 26 15:44 (141.9h) [2026-10-20 17:50:29][INFO][training.pretrain_engine] Starting global step 581 (accumulating 32 micro-batches)... [2026-10-20 17:50:29][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 17:50:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7403 [2026-10-20 17:50:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6129 [2026-10-20 17:50:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7736 [2026-10-20 17:50:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3222 [2026-10-20 17:51:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4444 [2026-10-20 17:51:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8599 [2026-10-20 17:51:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9192 [2026-10-20 17:51:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9228 [2026-10-20 17:51:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6633 [2026-10-20 17:51:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5922 [2026-10-20 17:51:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7387 [2026-10-20 17:51:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6197 [2026-10-20 17:51:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8998 [2026-10-20 17:51:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6354 [2026-10-20 17:51:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7360 [2026-10-20 17:51:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7998 [2026-10-20 17:51:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4663 [2026-10-20 17:51:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7288 [2026-10-20 17:51:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4818 [2026-10-20 17:51:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6389 [2026-10-20 17:52:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4465 [2026-10-20 17:52:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7087 [2026-10-20 17:52:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1067 [2026-10-20 17:52:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6653 [2026-10-20 17:52:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2704 [2026-10-20 17:52:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2834 [2026-10-20 17:52:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1328 [2026-10-20 17:52:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0000 [2026-10-20 17:52:48][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,524 | MFU: 1.8% [2026-10-20 17:52:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1780 [2026-10-20 17:52:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3683 [2026-10-20 17:52:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5096 [2026-10-20 17:52:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0449 [2026-10-20 17:52:53][INFO][training.pretrain_engine] [Ph2|Step 581] loss=5.7444 | ppl=312.45 | aux=0.02509 | lr=3.00e-04 | gnorm=0.617 | tok/s=29,253 | MFU=7.0% | step=143382ms | tokens=0.617B | mem=4.2/32.7GB | cost=$305.32/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 10:27 (136.6h) | Phase3: SKIPPED | SFT: Oct 26 15:12 (141.3h) | DPO (Final): Oct 26 15:46 (141.9h) [2026-10-20 17:52:53][INFO][training.pretrain_engine] Starting global step 582 (accumulating 32 micro-batches)... [2026-10-20 17:52:53][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 17:53:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3836 [2026-10-20 17:53:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8068 [2026-10-20 17:53:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3813 [2026-10-20 17:53:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3566 [2026-10-20 17:53:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7996 [2026-10-20 17:53:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2404 [2026-10-20 17:53:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3190 [2026-10-20 17:53:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6226 [2026-10-20 17:53:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4689 [2026-10-20 17:53:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8314 [2026-10-20 17:53:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0386 [2026-10-20 17:53:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8674 [2026-10-20 17:54:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9783 [2026-10-20 17:54:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0081 [2026-10-20 17:54:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1651 [2026-10-20 17:54:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8080 [2026-10-20 17:54:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6854 [2026-10-20 17:54:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7173 [2026-10-20 17:54:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8196 [2026-10-20 17:54:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5292 [2026-10-20 17:54:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1157 [2026-10-20 17:54:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3500 [2026-10-20 17:54:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3385 [2026-10-20 17:54:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7082 [2026-10-20 17:54:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9020 [2026-10-20 17:54:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3805 [2026-10-20 17:54:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8549 [2026-10-20 17:54:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9169 [2026-10-20 17:55:12][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,548 | MFU: 1.8% [2026-10-20 17:55:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8106 [2026-10-20 17:55:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1667 [2026-10-20 17:55:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6945 [2026-10-20 17:55:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6698 [2026-10-20 17:55:16][INFO][training.pretrain_engine] [Ph2|Step 582] loss=5.5979 | ppl=269.85 | aux=0.02508 | lr=3.00e-04 | gnorm=0.727 | tok/s=29,253 | MFU=7.0% | step=143380ms | tokens=0.618B | mem=4.2/32.7GB | cost=$305.85/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 10:29 (136.6h) | Phase3: SKIPPED | SFT: Oct 26 15:14 (141.3h) | DPO (Final): Oct 26 15:48 (141.9h) [2026-10-20 17:55:16][INFO][training.pretrain_engine] Starting global step 583 (accumulating 32 micro-batches)... [2026-10-20 17:55:16][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 17:55:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1463 [2026-10-20 17:55:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6492 [2026-10-20 17:55:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8449 [2026-10-20 17:55:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6965 [2026-10-20 17:55:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6832 [2026-10-20 17:55:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8116 [2026-10-20 17:55:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4866 [2026-10-20 17:55:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9402 [2026-10-20 17:56:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0794 [2026-10-20 17:56:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1128 [2026-10-20 17:56:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4555 [2026-10-20 17:56:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6951 [2026-10-20 17:56:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8785 [2026-10-20 17:56:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6071 [2026-10-20 17:56:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7231 [2026-10-20 17:56:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9133 [2026-10-20 17:56:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8505 [2026-10-20 17:56:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8236 [2026-10-20 17:56:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6405 [2026-10-20 17:56:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4257 [2026-10-20 17:57:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5015 [2026-10-20 17:57:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4581 [2026-10-20 17:57:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9723 [2026-10-20 17:57:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3369 [2026-10-20 17:57:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9214 [2026-10-20 17:57:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2997 [2026-10-20 17:57:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7597 [2026-10-20 17:57:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7119 [2026-10-20 17:57:35][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,578 | MFU: 1.8% [2026-10-20 17:57:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8038 [2026-10-20 17:57:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8185 [2026-10-20 17:57:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1407 [2026-10-20 17:57:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4547 [2026-10-20 17:57:40][INFO][training.pretrain_engine] [Ph2|Step 583] loss=5.6314 | ppl=279.05 | aux=0.02508 | lr=3.00e-04 | gnorm=1.188 | tok/s=29,253 | MFU=7.0% | step=143382ms | tokens=0.619B | mem=4.2/32.7GB | cost=$306.37/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 10:31 (136.6h) | Phase3: SKIPPED | SFT: Oct 26 15:16 (141.3h) | DPO (Final): Oct 26 15:50 (141.9h) [2026-10-20 17:57:40][INFO][training.pretrain_engine] Starting global step 584 (accumulating 32 micro-batches)... [2026-10-20 17:57:40][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 17:57:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3505 [2026-10-20 17:57:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5448 [2026-10-20 17:57:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0528 [2026-10-20 17:57:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3913 [2026-10-20 17:58:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9755 [2026-10-20 17:58:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1381 [2026-10-20 17:58:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0959 [2026-10-20 17:58:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0370 [2026-10-20 17:58:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6628 [2026-10-20 17:58:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1029 [2026-10-20 17:58:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2390 [2026-10-20 17:58:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2491 [2026-10-20 17:58:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3927 [2026-10-20 17:58:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0817 [2026-10-20 17:58:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1919 [2026-10-20 17:58:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1835 [2026-10-20 17:59:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9396 [2026-10-20 17:59:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1753 [2026-10-20 17:59:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6981 [2026-10-20 17:59:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7133 [2026-10-20 17:59:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1682 [2026-10-20 17:59:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5026 [2026-10-20 17:59:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1347 [2026-10-20 17:59:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5580 [2026-10-20 17:59:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8471 [2026-10-20 17:59:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6511 [2026-10-20 17:59:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1576 [2026-10-20 17:59:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6280 [2026-10-20 17:59:59][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,526 | MFU: 1.8% [2026-10-20 18:00:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9022 [2026-10-20 18:00:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9894 [2026-10-20 18:00:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8734 [2026-10-20 18:00:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4957 [2026-10-20 18:00:04][INFO][training.pretrain_engine] [Ph2|Step 584] loss=5.6098 | ppl=273.10 | aux=0.02509 | lr=3.00e-04 | gnorm=1.492 | tok/s=29,251 | MFU=7.0% | step=143391ms | tokens=0.620B | mem=4.2/32.7GB | cost=$306.90/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 10:33 (136.6h) | Phase3: SKIPPED | SFT: Oct 26 15:18 (141.3h) | DPO (Final): Oct 26 15:52 (141.9h) [2026-10-20 18:00:04][INFO][training.pretrain_engine] Starting global step 585 (accumulating 32 micro-batches)... [2026-10-20 18:00:04][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 18:00:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2047 [2026-10-20 18:00:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4648 [2026-10-20 18:00:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7384 [2026-10-20 18:00:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7290 [2026-10-20 18:00:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8705 [2026-10-20 18:00:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2979 [2026-10-20 18:00:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7579 [2026-10-20 18:00:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9624 [2026-10-20 18:00:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7803 [2026-10-20 18:00:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1068 [2026-10-20 18:00:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2941 [2026-10-20 18:00:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8361 [2026-10-20 18:01:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9160 [2026-10-20 18:01:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2106 [2026-10-20 18:01:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8401 [2026-10-20 18:01:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0853 [2026-10-20 18:01:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8339 [2026-10-20 18:01:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4636 [2026-10-20 18:01:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1148 [2026-10-20 18:01:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1894 [2026-10-20 18:01:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1895 [2026-10-20 18:01:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7613 [2026-10-20 18:01:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9195 [2026-10-20 18:01:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6381 [2026-10-20 18:02:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7814 [2026-10-20 18:02:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7633 [2026-10-20 18:02:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1442 [2026-10-20 18:02:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7703 [2026-10-20 18:02:23][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,529 | MFU: 1.8% [2026-10-20 18:02:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0372 [2026-10-20 18:02:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9203 [2026-10-20 18:02:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9141 [2026-10-20 18:02:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9206 [2026-10-20 18:02:27][INFO][training.pretrain_engine] [Ph2|Step 585] loss=5.6787 | ppl=292.56 | aux=0.02510 | lr=3.00e-04 | gnorm=0.754 | tok/s=29,250 | MFU=7.0% | step=143395ms | tokens=0.621B | mem=4.2/32.7GB | cost=$307.42/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 10:35 (136.6h) | Phase3: SKIPPED | SFT: Oct 26 15:20 (141.3h) | DPO (Final): Oct 26 15:54 (141.9h) [2026-10-20 18:02:27][INFO][training.pretrain_engine] Starting global step 586 (accumulating 32 micro-batches)... [2026-10-20 18:02:27][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 18:02:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9744 [2026-10-20 18:02:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7931 [2026-10-20 18:02:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5616 [2026-10-20 18:02:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9514 [2026-10-20 18:03:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3064 [2026-10-20 18:03:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7383 [2026-10-20 18:03:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4831 [2026-10-20 18:03:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9182 [2026-10-20 18:03:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8416 [2026-10-20 18:03:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8112 [2026-10-20 18:03:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8255 [2026-10-20 18:03:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7589 [2026-10-20 18:03:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8478 [2026-10-20 18:03:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5214 [2026-10-20 18:03:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8395 [2026-10-20 18:03:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7849 [2026-10-20 18:03:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3435 [2026-10-20 18:03:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7830 [2026-10-20 18:03:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1693 [2026-10-20 18:03:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5236 [2026-10-20 18:04:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7132 [2026-10-20 18:04:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5825 [2026-10-20 18:04:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7384 [2026-10-20 18:04:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0357 [2026-10-20 18:04:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5666 [2026-10-20 18:04:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0851 [2026-10-20 18:04:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4946 [2026-10-20 18:04:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5188 [2026-10-20 18:04:46][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,556 | MFU: 1.8% [2026-10-20 18:04:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3833 [2026-10-20 18:04:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5962 [2026-10-20 18:04:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7751 [2026-10-20 18:04:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7113 [2026-10-20 18:04:51][INFO][training.pretrain_engine] [Ph2|Step 586] loss=5.8000 | ppl=330.31 | aux=0.02509 | lr=3.00e-04 | gnorm=1.523 | tok/s=29,253 | MFU=7.0% | step=143379ms | tokens=0.622B | mem=4.2/32.7GB | cost=$307.95/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 10:36 (136.5h) | Phase3: SKIPPED | SFT: Oct 26 15:21 (141.3h) | DPO (Final): Oct 26 15:55 (141.8h) [2026-10-20 18:04:51][INFO][training.pretrain_engine] Starting global step 587 (accumulating 32 micro-batches)... [2026-10-20 18:04:51][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 18:05:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2082 [2026-10-20 18:05:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4569 [2026-10-20 18:05:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7030 [2026-10-20 18:05:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0361 [2026-10-20 18:05:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7245 [2026-10-20 18:05:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6117 [2026-10-20 18:05:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7170 [2026-10-20 18:05:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3118 [2026-10-20 18:05:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9246 [2026-10-20 18:05:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7978 [2026-10-20 18:05:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1563 [2026-10-20 18:05:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1407 [2026-10-20 18:06:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9459 [2026-10-20 18:06:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7395 [2026-10-20 18:06:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5860 [2026-10-20 18:06:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9400 [2026-10-20 18:06:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4739 [2026-10-20 18:06:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6989 [2026-10-20 18:06:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6134 [2026-10-20 18:06:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5608 [2026-10-20 18:06:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6962 [2026-10-20 18:06:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5108 [2026-10-20 18:06:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6534 [2026-10-20 18:06:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9525 [2026-10-20 18:06:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4371 [2026-10-20 18:06:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7827 [2026-10-20 18:06:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1600 [2026-10-20 18:06:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0423 [2026-10-20 18:07:10][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,532 | MFU: 1.8% [2026-10-20 18:07:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0902 [2026-10-20 18:07:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3715 [2026-10-20 18:07:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1239 [2026-10-20 18:07:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6237 [2026-10-20 18:07:15][INFO][training.pretrain_engine] [Ph2|Step 587] loss=5.6200 | ppl=275.88 | aux=0.02509 | lr=3.00e-04 | gnorm=0.805 | tok/s=29,257 | MFU=7.0% | step=143359ms | tokens=0.623B | mem=4.2/32.7GB | cost=$308.48/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 10:37 (136.5h) | Phase3: SKIPPED | SFT: Oct 26 15:22 (141.2h) | DPO (Final): Oct 26 15:56 (141.8h) [2026-10-20 18:07:18][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step587 | Tokens: 0.623B | Spent: $308.49 | Remaining: $91.51 [2026-10-20 18:07:18][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step587 [2026-10-20 18:07:18][INFO][training.pretrain_engine] Starting global step 588 (accumulating 32 micro-batches)... [2026-10-20 18:07:18][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 18:07:18][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 18:07:18][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 18:07:18][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 18:07:18][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step587 to Phase-Technologies/forge-500m-pretrain (path: phase2_step587)... [2026-10-20 18:07:18][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 18:07:28][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 18:07:28][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step587 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 18:07:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0962 [2026-10-20 18:07:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6447 [2026-10-20 18:07:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8858 [2026-10-20 18:07:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5314 [2026-10-20 18:07:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6752 [2026-10-20 18:07:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6861 [2026-10-20 18:07:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3665 [2026-10-20 18:07:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7416 [2026-10-20 18:08:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6642 [2026-10-20 18:08:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7394 [2026-10-20 18:08:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4189 [2026-10-20 18:08:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3711 [2026-10-20 18:08:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4008 [2026-10-20 18:08:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6414 [2026-10-20 18:08:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7818 [2026-10-20 18:08:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8911 [2026-10-20 18:08:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7123 [2026-10-20 18:08:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3624 [2026-10-20 18:08:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6237 [2026-10-20 18:08:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6495 [2026-10-20 18:09:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6316 [2026-10-20 18:09:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7937 [2026-10-20 18:09:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6398 [2026-10-20 18:09:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9019 [2026-10-20 18:09:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5337 [2026-10-20 18:09:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8255 [2026-10-20 18:09:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6093 [2026-10-20 18:09:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5940 [2026-10-20 18:09:37][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,540 | MFU: 1.8% [2026-10-20 18:09:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7082 [2026-10-20 18:09:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2209 [2026-10-20 18:09:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7576 [2026-10-20 18:09:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0058 [2026-10-20 18:09:42][INFO][training.pretrain_engine] [Ph2|Step 588] loss=5.6468 | ppl=283.38 | aux=0.02507 | lr=3.00e-04 | gnorm=1.195 | tok/s=29,257 | MFU=7.0% | step=143362ms | tokens=0.624B | mem=4.2/32.7GB | cost=$309.01/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 10:39 (136.5h) | Phase3: SKIPPED | SFT: Oct 26 15:24 (141.2h) | DPO (Final): Oct 26 15:58 (141.8h) [2026-10-20 18:09:42][INFO][training.pretrain_engine] Starting global step 589 (accumulating 32 micro-batches)... [2026-10-20 18:09:42][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 18:10:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7298 [2026-10-20 18:10:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4706 [2026-10-20 18:10:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7784 [2026-10-20 18:10:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3492 [2026-10-20 18:10:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1723 [2026-10-20 18:10:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9080 [2026-10-20 18:10:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9463 [2026-10-20 18:10:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8484 [2026-10-20 18:10:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1960 [2026-10-20 18:10:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3641 [2026-10-20 18:10:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1397 [2026-10-20 18:10:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6643 [2026-10-20 18:10:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5199 [2026-10-20 18:10:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4915 [2026-10-20 18:10:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7855 [2026-10-20 18:10:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6493 [2026-10-20 18:11:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1765 [2026-10-20 18:11:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1943 [2026-10-20 18:11:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3278 [2026-10-20 18:11:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3267 [2026-10-20 18:11:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1956 [2026-10-20 18:11:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5808 [2026-10-20 18:11:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7684 [2026-10-20 18:11:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5236 [2026-10-20 18:11:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9142 [2026-10-20 18:11:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0411 [2026-10-20 18:11:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8795 [2026-10-20 18:11:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9819 [2026-10-20 18:12:00][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,569 | MFU: 1.8% [2026-10-20 18:12:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7996 [2026-10-20 18:12:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7624 [2026-10-20 18:12:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8705 [2026-10-20 18:12:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8483 [2026-10-20 18:12:06][INFO][training.pretrain_engine] [Ph2|Step 589] loss=5.6290 | ppl=278.38 | aux=0.02508 | lr=3.00e-04 | gnorm=0.898 | tok/s=29,255 | MFU=7.0% | step=143370ms | tokens=0.625B | mem=4.2/32.7GB | cost=$309.54/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 10:41 (136.5h) | Phase3: SKIPPED | SFT: Oct 26 15:26 (141.2h) | DPO (Final): Oct 26 16:00 (141.8h) [2026-10-20 18:12:06][INFO][training.pretrain_engine] Starting global step 590 (accumulating 32 micro-batches)... [2026-10-20 18:12:06][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 18:12:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7922 [2026-10-20 18:12:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5644 [2026-10-20 18:12:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7152 [2026-10-20 18:12:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8906 [2026-10-20 18:12:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8196 [2026-10-20 18:12:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8401 [2026-10-20 18:12:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6865 [2026-10-20 18:12:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3577 [2026-10-20 18:12:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2823 [2026-10-20 18:12:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3805 [2026-10-20 18:13:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8826 [2026-10-20 18:13:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6865 [2026-10-20 18:13:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7422 [2026-10-20 18:13:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1730 [2026-10-20 18:13:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7085 [2026-10-20 18:13:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3302 [2026-10-20 18:13:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5155 [2026-10-20 18:13:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7165 [2026-10-20 18:13:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6221 [2026-10-20 18:13:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7370 [2026-10-20 18:13:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8732 [2026-10-20 18:13:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8925 [2026-10-20 18:13:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7369 [2026-10-20 18:13:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0150 [2026-10-20 18:14:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7415 [2026-10-20 18:14:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5781 [2026-10-20 18:14:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4850 [2026-10-20 18:14:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6457 [2026-10-20 18:14:24][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,577 | MFU: 1.8% [2026-10-20 18:14:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8362 [2026-10-20 18:14:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9946 [2026-10-20 18:14:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0888 [2026-10-20 18:14:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4566 [2026-10-20 18:14:29][INFO][training.pretrain_engine] [Ph2|Step 590] loss=5.6879 | ppl=295.26 | aux=0.02508 | lr=3.00e-04 | gnorm=1.000 | tok/s=29,254 | MFU=7.0% | step=143376ms | tokens=0.626B | mem=4.2/32.7GB | cost=$310.06/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 10:43 (136.5h) | Phase3: SKIPPED | SFT: Oct 26 15:28 (141.2h) | DPO (Final): Oct 26 16:02 (141.8h) [2026-10-20 18:14:29][INFO][training.pretrain_engine] Starting global step 591 (accumulating 32 micro-batches)... [2026-10-20 18:14:29][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 18:14:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7841 [2026-10-20 18:14:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6222 [2026-10-20 18:14:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8698 [2026-10-20 18:14:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8350 [2026-10-20 18:15:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6659 [2026-10-20 18:15:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7069 [2026-10-20 18:15:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8603 [2026-10-20 18:15:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6330 [2026-10-20 18:15:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8181 [2026-10-20 18:15:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2160 [2026-10-20 18:15:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1978 [2026-10-20 18:15:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0110 [2026-10-20 18:15:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4051 [2026-10-20 18:15:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7788 [2026-10-20 18:15:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3070 [2026-10-20 18:15:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0941 [2026-10-20 18:15:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1280 [2026-10-20 18:15:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5458 [2026-10-20 18:15:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1884 [2026-10-20 18:15:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8487 [2026-10-20 18:16:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0032 [2026-10-20 18:16:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7704 [2026-10-20 18:16:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6219 [2026-10-20 18:16:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6833 [2026-10-20 18:16:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3088 [2026-10-20 18:16:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4086 [2026-10-20 18:16:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2620 [2026-10-20 18:16:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1422 [2026-10-20 18:16:48][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,548 | MFU: 1.8% [2026-10-20 18:16:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6377 [2026-10-20 18:16:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9822 [2026-10-20 18:16:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7984 [2026-10-20 18:16:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8649 [2026-10-20 18:16:53][INFO][training.pretrain_engine] [Ph2|Step 591] loss=5.6468 | ppl=283.39 | aux=0.02510 | lr=3.00e-04 | gnorm=1.023 | tok/s=29,253 | MFU=7.0% | step=143378ms | tokens=0.627B | mem=4.2/32.7GB | cost=$310.59/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 10:45 (136.5h) | Phase3: SKIPPED | SFT: Oct 26 15:30 (141.2h) | DPO (Final): Oct 26 16:04 (141.8h) [2026-10-20 18:16:53][INFO][training.pretrain_engine] Starting global step 592 (accumulating 32 micro-batches)... [2026-10-20 18:16:53][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 18:17:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0558 [2026-10-20 18:17:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0777 [2026-10-20 18:17:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9498 [2026-10-20 18:17:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8339 [2026-10-20 18:17:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3833 [2026-10-20 18:17:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0050 [2026-10-20 18:17:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2201 [2026-10-20 18:17:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4074 [2026-10-20 18:17:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1985 [2026-10-20 18:17:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7465 [2026-10-20 18:17:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6236 [2026-10-20 18:17:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8869 [2026-10-20 18:18:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6022 [2026-10-20 18:18:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8731 [2026-10-20 18:18:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3544 [2026-10-20 18:18:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2292 [2026-10-20 18:18:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3525 [2026-10-20 18:18:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5091 [2026-10-20 18:18:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1368 [2026-10-20 18:18:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3529 [2026-10-20 18:18:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5148 [2026-10-20 18:18:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6837 [2026-10-20 18:18:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7553 [2026-10-20 18:18:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2842 [2026-10-20 18:18:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8572 [2026-10-20 18:18:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7366 [2026-10-20 18:18:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8943 [2026-10-20 18:18:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0242 [2026-10-20 18:19:12][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,559 | MFU: 1.8% [2026-10-20 18:19:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7935 [2026-10-20 18:19:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7023 [2026-10-20 18:19:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1014 [2026-10-20 18:19:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9708 [2026-10-20 18:19:17][INFO][training.pretrain_engine] [Ph2|Step 592] loss=5.7087 | ppl=301.46 | aux=0.02509 | lr=3.00e-04 | gnorm=1.195 | tok/s=29,252 | MFU=7.0% | step=143384ms | tokens=0.628B | mem=4.2/32.7GB | cost=$311.11/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 10:47 (136.5h) | Phase3: SKIPPED | SFT: Oct 26 15:32 (141.2h) | DPO (Final): Oct 26 16:06 (141.8h) [2026-10-20 18:19:17][INFO][training.pretrain_engine] Starting global step 593 (accumulating 32 micro-batches)... [2026-10-20 18:19:17][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 18:19:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6341 [2026-10-20 18:19:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9193 [2026-10-20 18:19:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6201 [2026-10-20 18:19:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4759 [2026-10-20 18:19:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4719 [2026-10-20 18:19:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3967 [2026-10-20 18:19:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9546 [2026-10-20 18:19:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.6867 [2026-10-20 18:20:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4423 [2026-10-20 18:20:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7854 [2026-10-20 18:20:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5457 [2026-10-20 18:20:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6471 [2026-10-20 18:20:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8055 [2026-10-20 18:20:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7908 [2026-10-20 18:20:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6746 [2026-10-20 18:20:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9746 [2026-10-20 18:20:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0411 [2026-10-20 18:20:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7855 [2026-10-20 18:20:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7970 [2026-10-20 18:20:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7499 [2026-10-20 18:21:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5767 [2026-10-20 18:21:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7611 [2026-10-20 18:21:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0754 [2026-10-20 18:21:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8744 [2026-10-20 18:21:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5711 [2026-10-20 18:21:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7574 [2026-10-20 18:21:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9020 [2026-10-20 18:21:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7339 [2026-10-20 18:21:35][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,573 | MFU: 1.8% [2026-10-20 18:21:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7480 [2026-10-20 18:21:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7011 [2026-10-20 18:21:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7125 [2026-10-20 18:21:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6671 [2026-10-20 18:21:40][INFO][training.pretrain_engine] [Ph2|Step 593] loss=5.7900 | ppl=327.03 | aux=0.02507 | lr=3.00e-04 | gnorm=0.723 | tok/s=29,252 | MFU=7.0% | step=143387ms | tokens=0.629B | mem=4.2/32.7GB | cost=$311.64/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 10:49 (136.5h) | Phase3: SKIPPED | SFT: Oct 26 15:34 (141.2h) | DPO (Final): Oct 26 16:08 (141.8h) [2026-10-20 18:21:40][INFO][training.pretrain_engine] Starting global step 594 (accumulating 32 micro-batches)... [2026-10-20 18:21:40][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 18:21:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7957 [2026-10-20 18:21:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7100 [2026-10-20 18:21:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6894 [2026-10-20 18:21:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6516 [2026-10-20 18:22:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2071 [2026-10-20 18:22:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7471 [2026-10-20 18:22:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2997 [2026-10-20 18:22:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3031 [2026-10-20 18:22:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6057 [2026-10-20 18:22:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7235 [2026-10-20 18:22:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5139 [2026-10-20 18:22:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3995 [2026-10-20 18:22:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8232 [2026-10-20 18:22:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3418 [2026-10-20 18:22:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6123 [2026-10-20 18:22:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7893 [2026-10-20 18:23:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1566 [2026-10-20 18:23:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5651 [2026-10-20 18:23:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2263 [2026-10-20 18:23:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3934 [2026-10-20 18:23:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4860 [2026-10-20 18:23:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7350 [2026-10-20 18:23:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4491 [2026-10-20 18:23:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4783 [2026-10-20 18:23:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8523 [2026-10-20 18:23:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2997 [2026-10-20 18:23:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4428 [2026-10-20 18:23:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8518 [2026-10-20 18:23:58][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,567 | MFU: 1.8% [2026-10-20 18:24:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6347 [2026-10-20 18:24:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9717 [2026-10-20 18:24:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6921 [2026-10-20 18:24:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8038 [2026-10-20 18:24:03][INFO][training.pretrain_engine] [Ph2|Step 594] loss=5.6603 | ppl=287.24 | aux=0.02509 | lr=3.00e-04 | gnorm=1.016 | tok/s=29,251 | MFU=7.0% | step=143392ms | tokens=0.630B | mem=4.2/32.7GB | cost=$312.16/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 10:51 (136.5h) | Phase3: SKIPPED | SFT: Oct 26 15:36 (141.2h) | DPO (Final): Oct 26 16:10 (141.8h) [2026-10-20 18:24:03][INFO][training.pretrain_engine] Starting global step 595 (accumulating 32 micro-batches)... [2026-10-20 18:24:03][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 18:24:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7057 [2026-10-20 18:24:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4343 [2026-10-20 18:24:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5907 [2026-10-20 18:24:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5920 [2026-10-20 18:24:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9690 [2026-10-20 18:24:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0617 [2026-10-20 18:24:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7725 [2026-10-20 18:24:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8308 [2026-10-20 18:24:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5984 [2026-10-20 18:24:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3465 [2026-10-20 18:24:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3126 [2026-10-20 18:24:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2782 [2026-10-20 18:25:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9570 [2026-10-20 18:25:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4018 [2026-10-20 18:25:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4764 [2026-10-20 18:25:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3547 [2026-10-20 18:25:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9486 [2026-10-20 18:25:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9933 [2026-10-20 18:25:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7339 [2026-10-20 18:25:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4123 [2026-10-20 18:25:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8258 [2026-10-20 18:25:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9471 [2026-10-20 18:25:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8572 [2026-10-20 18:25:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9592 [2026-10-20 18:26:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8373 [2026-10-20 18:26:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0260 [2026-10-20 18:26:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3381 [2026-10-20 18:26:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7308 [2026-10-20 18:26:23][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,533 | MFU: 1.8% [2026-10-20 18:26:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2717 [2026-10-20 18:26:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7250 [2026-10-20 18:26:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6265 [2026-10-20 18:26:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4702 [2026-10-20 18:26:27][INFO][training.pretrain_engine] [Ph2|Step 595] loss=5.7093 | ppl=301.65 | aux=0.02510 | lr=3.00e-04 | gnorm=0.941 | tok/s=29,250 | MFU=7.0% | step=143397ms | tokens=0.631B | mem=4.2/32.7GB | cost=$312.69/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 10:53 (136.5h) | Phase3: SKIPPED | SFT: Oct 26 15:38 (141.2h) | DPO (Final): Oct 26 16:12 (141.8h) [2026-10-20 18:26:27][INFO][training.pretrain_engine] Starting global step 596 (accumulating 32 micro-batches)... [2026-10-20 18:26:27][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 18:26:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6696 [2026-10-20 18:26:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8696 [2026-10-20 18:26:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9839 [2026-10-20 18:26:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3606 [2026-10-20 18:27:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7760 [2026-10-20 18:27:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8220 [2026-10-20 18:27:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4819 [2026-10-20 18:27:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3191 [2026-10-20 18:27:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7523 [2026-10-20 18:27:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8169 [2026-10-20 18:27:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5144 [2026-10-20 18:27:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6603 [2026-10-20 18:27:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8366 [2026-10-20 18:27:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1960 [2026-10-20 18:27:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1036 [2026-10-20 18:27:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8585 [2026-10-20 18:27:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2036 [2026-10-20 18:27:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7143 [2026-10-20 18:27:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4050 [2026-10-20 18:27:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5289 [2026-10-20 18:28:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5334 [2026-10-20 18:28:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3952 [2026-10-20 18:28:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8425 [2026-10-20 18:28:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7435 [2026-10-20 18:28:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6205 [2026-10-20 18:28:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2696 [2026-10-20 18:28:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5499 [2026-10-20 18:28:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8219 [2026-10-20 18:28:46][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,549 | MFU: 1.8% [2026-10-20 18:28:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5616 [2026-10-20 18:28:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5396 [2026-10-20 18:28:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5848 [2026-10-20 18:28:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9008 [2026-10-20 18:28:51][INFO][training.pretrain_engine] [Ph2|Step 596] loss=5.6824 | ppl=293.64 | aux=0.02508 | lr=3.00e-04 | gnorm=1.109 | tok/s=29,249 | MFU=7.0% | step=143402ms | tokens=0.632B | mem=4.2/32.7GB | cost=$313.21/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 10:55 (136.5h) | Phase3: SKIPPED | SFT: Oct 26 15:40 (141.2h) | DPO (Final): Oct 26 16:15 (141.8h) [2026-10-20 18:28:54][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step596 | Tokens: 0.632B | Spent: $313.22 | Remaining: $86.78 [2026-10-20 18:28:54][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step596 [2026-10-20 18:28:54][INFO][training.pretrain_engine] Starting global step 597 (accumulating 32 micro-batches)... [2026-10-20 18:28:54][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 18:28:54][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 18:28:54][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 18:28:54][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 18:28:54][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step596 to Phase-Technologies/forge-500m-pretrain (path: phase2_step596)... [2026-10-20 18:28:54][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 18:29:06][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 18:29:06][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step596 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 18:29:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4044 [2026-10-20 18:29:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4029 [2026-10-20 18:29:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1639 [2026-10-20 18:29:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5967 [2026-10-20 18:29:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6271 [2026-10-20 18:29:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8419 [2026-10-20 18:29:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1917 [2026-10-20 18:29:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8451 [2026-10-20 18:29:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8758 [2026-10-20 18:29:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6736 [2026-10-20 18:29:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2715 [2026-10-20 18:29:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8357 [2026-10-20 18:30:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6790 [2026-10-20 18:30:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8598 [2026-10-20 18:30:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8893 [2026-10-20 18:30:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8878 [2026-10-20 18:30:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0204 [2026-10-20 18:30:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5022 [2026-10-20 18:30:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9534 [2026-10-20 18:30:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2671 [2026-10-20 18:30:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9147 [2026-10-20 18:30:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6246 [2026-10-20 18:30:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9441 [2026-10-20 18:30:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0512 [2026-10-20 18:30:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8520 [2026-10-20 18:30:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7063 [2026-10-20 18:30:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7175 [2026-10-20 18:31:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1167 [2026-10-20 18:31:13][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,537 | MFU: 1.8% [2026-10-20 18:31:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4344 [2026-10-20 18:31:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2667 [2026-10-20 18:31:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5250 [2026-10-20 18:31:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5717 [2026-10-20 18:31:18][INFO][training.pretrain_engine] [Ph2|Step 597] loss=5.7956 | ppl=328.86 | aux=0.02509 | lr=3.00e-04 | gnorm=1.000 | tok/s=29,248 | MFU=7.0% | step=143406ms | tokens=0.633B | mem=4.2/32.7GB | cost=$313.75/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 10:58 (136.4h) | Phase3: SKIPPED | SFT: Oct 26 15:42 (141.2h) | DPO (Final): Oct 26 16:17 (141.8h) [2026-10-20 18:31:18][INFO][training.pretrain_engine] Starting global step 598 (accumulating 32 micro-batches)... [2026-10-20 18:31:18][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 18:31:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7293 [2026-10-20 18:31:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.6966 [2026-10-20 18:31:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7038 [2026-10-20 18:31:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2541 [2026-10-20 18:31:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5140 [2026-10-20 18:31:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5266 [2026-10-20 18:31:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.8101 [2026-10-20 18:31:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5017 [2026-10-20 18:32:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7094 [2026-10-20 18:32:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8781 [2026-10-20 18:32:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9255 [2026-10-20 18:32:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4774 [2026-10-20 18:32:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9391 [2026-10-20 18:32:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8145 [2026-10-20 18:32:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5303 [2026-10-20 18:32:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5800 [2026-10-20 18:32:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1590 [2026-10-20 18:32:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4632 [2026-10-20 18:32:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6749 [2026-10-20 18:32:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4497 [2026-10-20 18:33:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6423 [2026-10-20 18:33:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6072 [2026-10-20 18:33:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5603 [2026-10-20 18:33:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1671 [2026-10-20 18:33:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0715 [2026-10-20 18:33:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0636 [2026-10-20 18:33:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5715 [2026-10-20 18:33:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7114 [2026-10-20 18:33:37][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,531 | MFU: 1.8% [2026-10-20 18:33:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6902 [2026-10-20 18:33:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2685 [2026-10-20 18:33:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9953 [2026-10-20 18:33:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7051 [2026-10-20 18:33:42][INFO][training.pretrain_engine] [Ph2|Step 598] loss=5.5861 | ppl=266.70 | aux=0.02509 | lr=3.00e-04 | gnorm=0.875 | tok/s=29,247 | MFU=7.0% | step=143408ms | tokens=0.634B | mem=4.2/32.7GB | cost=$314.28/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 10:59 (136.4h) | Phase3: SKIPPED | SFT: Oct 26 15:44 (141.2h) | DPO (Final): Oct 26 16:19 (141.8h) [2026-10-20 18:33:42][INFO][training.pretrain_engine] Starting global step 599 (accumulating 32 micro-batches)... [2026-10-20 18:33:42][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 18:33:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1166 [2026-10-20 18:34:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5089 [2026-10-20 18:34:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8473 [2026-10-20 18:34:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3242 [2026-10-20 18:34:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3390 [2026-10-20 18:34:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9726 [2026-10-20 18:34:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9327 [2026-10-20 18:34:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0011 [2026-10-20 18:34:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9473 [2026-10-20 18:34:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3395 [2026-10-20 18:34:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1036 [2026-10-20 18:34:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6113 [2026-10-20 18:34:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5481 [2026-10-20 18:34:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3713 [2026-10-20 18:34:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6555 [2026-10-20 18:34:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0539 [2026-10-20 18:35:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7283 [2026-10-20 18:35:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5441 [2026-10-20 18:35:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9841 [2026-10-20 18:35:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1497 [2026-10-20 18:35:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9440 [2026-10-20 18:35:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9442 [2026-10-20 18:35:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1942 [2026-10-20 18:35:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0134 [2026-10-20 18:35:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6165 [2026-10-20 18:35:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3871 [2026-10-20 18:35:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7924 [2026-10-20 18:35:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5093 [2026-10-20 18:36:01][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,551 | MFU: 1.8% [2026-10-20 18:36:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9217 [2026-10-20 18:36:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8190 [2026-10-20 18:36:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6608 [2026-10-20 18:36:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4782 [2026-10-20 18:36:05][INFO][training.pretrain_engine] [Ph2|Step 599] loss=5.6454 | ppl=282.98 | aux=0.02511 | lr=3.00e-04 | gnorm=0.883 | tok/s=29,247 | MFU=7.0% | step=143410ms | tokens=0.635B | mem=4.2/32.7GB | cost=$314.80/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 11:01 (136.4h) | Phase3: SKIPPED | SFT: Oct 26 15:46 (141.2h) | DPO (Final): Oct 26 16:21 (141.7h) [2026-10-20 18:36:05][INFO][training.pretrain_engine] Starting global step 600 (accumulating 32 micro-batches)... [2026-10-20 18:36:05][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 18:36:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3716 [2026-10-20 18:36:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5073 [2026-10-20 18:36:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6472 [2026-10-20 18:36:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7732 [2026-10-20 18:36:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4399 [2026-10-20 18:36:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9694 [2026-10-20 18:36:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9137 [2026-10-20 18:36:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1566 [2026-10-20 18:36:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3216 [2026-10-20 18:36:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6810 [2026-10-20 18:36:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3487 [2026-10-20 18:36:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4117 [2026-10-20 18:37:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6316 [2026-10-20 18:37:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9020 [2026-10-20 18:37:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2484 [2026-10-20 18:37:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3673 [2026-10-20 18:37:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9939 [2026-10-20 18:37:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7766 [2026-10-20 18:37:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8135 [2026-10-20 18:37:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8297 [2026-10-20 18:37:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4233 [2026-10-20 18:37:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7720 [2026-10-20 18:37:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4860 [2026-10-20 18:37:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1462 [2026-10-20 18:38:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7809 [2026-10-20 18:38:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4395 [2026-10-20 18:38:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8098 [2026-10-20 18:38:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7172 [2026-10-20 18:38:24][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,564 | MFU: 1.8% [2026-10-20 18:38:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8085 [2026-10-20 18:38:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8250 [2026-10-20 18:38:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1062 [2026-10-20 18:38:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7297 [2026-10-20 18:38:29][INFO][training.pretrain_engine] [Ph2|Step 600] loss=5.6536 | ppl=285.33 | aux=0.02509 | lr=3.00e-04 | gnorm=0.848 | tok/s=29,246 | MFU=7.0% | step=143414ms | tokens=0.636B | mem=4.2/32.7GB | cost=$315.33/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 11:03 (136.4h) | Phase3: SKIPPED | SFT: Oct 26 15:48 (141.2h) | DPO (Final): Oct 26 16:23 (141.7h) [2026-10-20 18:38:29][INFO][training.pretrain_engine] Starting global step 601 (accumulating 32 micro-batches)... [2026-10-20 18:38:29][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 18:38:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1040 [2026-10-20 18:38:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0845 [2026-10-20 18:38:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6928 [2026-10-20 18:38:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6956 [2026-10-20 18:39:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4466 [2026-10-20 18:39:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3585 [2026-10-20 18:39:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6647 [2026-10-20 18:39:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6679 [2026-10-20 18:39:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5625 [2026-10-20 18:39:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6537 [2026-10-20 18:39:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8623 [2026-10-20 18:39:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1981 [2026-10-20 18:39:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8168 [2026-10-20 18:39:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4833 [2026-10-20 18:39:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8206 [2026-10-20 18:39:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8664 [2026-10-20 18:39:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2401 [2026-10-20 18:39:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8041 [2026-10-20 18:39:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1112 [2026-10-20 18:39:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3285 [2026-10-20 18:40:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6151 [2026-10-20 18:40:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7470 [2026-10-20 18:40:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5270 [2026-10-20 18:40:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4396 [2026-10-20 18:40:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4500 [2026-10-20 18:40:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6611 [2026-10-20 18:40:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6846 [2026-10-20 18:40:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1512 [2026-10-20 18:40:48][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,522 | MFU: 1.8% [2026-10-20 18:40:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6836 [2026-10-20 18:40:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8002 [2026-10-20 18:40:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8032 [2026-10-20 18:40:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1148 [2026-10-20 18:40:53][INFO][training.pretrain_engine] [Ph2|Step 601] loss=5.6514 | ppl=284.68 | aux=0.02509 | lr=3.00e-04 | gnorm=0.691 | tok/s=29,244 | MFU=7.0% | step=143423ms | tokens=0.638B | mem=4.2/32.7GB | cost=$315.85/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 11:06 (136.4h) | Phase3: SKIPPED | SFT: Oct 26 15:51 (141.2h) | DPO (Final): Oct 26 16:25 (141.7h) [2026-10-20 18:40:53][INFO][training.pretrain_engine] Starting global step 602 (accumulating 32 micro-batches)... [2026-10-20 18:40:53][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 18:41:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0377 [2026-10-20 18:41:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8089 [2026-10-20 18:41:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5847 [2026-10-20 18:41:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1923 [2026-10-20 18:41:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2093 [2026-10-20 18:41:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6686 [2026-10-20 18:41:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8981 [2026-10-20 18:41:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6785 [2026-10-20 18:41:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8239 [2026-10-20 18:41:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6869 [2026-10-20 18:41:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4398 [2026-10-20 18:41:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1514 [2026-10-20 18:42:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7853 [2026-10-20 18:42:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7505 [2026-10-20 18:42:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6017 [2026-10-20 18:42:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7525 [2026-10-20 18:42:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9080 [2026-10-20 18:42:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8932 [2026-10-20 18:42:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4042 [2026-10-20 18:42:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6851 [2026-10-20 18:42:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7767 [2026-10-20 18:42:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6403 [2026-10-20 18:42:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9713 [2026-10-20 18:42:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9979 [2026-10-20 18:42:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4879 [2026-10-20 18:42:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5572 [2026-10-20 18:42:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7148 [2026-10-20 18:42:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4457 [2026-10-20 18:43:12][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,520 | MFU: 1.8% [2026-10-20 18:43:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0360 [2026-10-20 18:43:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9141 [2026-10-20 18:43:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9296 [2026-10-20 18:43:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9936 [2026-10-20 18:43:17][INFO][training.pretrain_engine] [Ph2|Step 602] loss=5.6530 | ppl=285.16 | aux=0.02509 | lr=3.00e-04 | gnorm=0.730 | tok/s=29,242 | MFU=7.0% | step=143432ms | tokens=0.639B | mem=4.2/32.7GB | cost=$316.38/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 11:08 (136.4h) | Phase3: SKIPPED | SFT: Oct 26 15:53 (141.2h) | DPO (Final): Oct 26 16:27 (141.7h) [2026-10-20 18:43:17][INFO][training.pretrain_engine] Starting global step 603 (accumulating 32 micro-batches)... [2026-10-20 18:43:17][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 18:43:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3620 [2026-10-20 18:43:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6004 [2026-10-20 18:43:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7718 [2026-10-20 18:43:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3035 [2026-10-20 18:43:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5575 [2026-10-20 18:43:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2468 [2026-10-20 18:43:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3984 [2026-10-20 18:43:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1584 [2026-10-20 18:44:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6743 [2026-10-20 18:44:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6965 [2026-10-20 18:44:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0510 [2026-10-20 18:44:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4198 [2026-10-20 18:44:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5513 [2026-10-20 18:44:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8246 [2026-10-20 18:44:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5625 [2026-10-20 18:44:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5046 [2026-10-20 18:44:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5629 [2026-10-20 18:44:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5782 [2026-10-20 18:44:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5598 [2026-10-20 18:44:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.2241 [2026-10-20 18:45:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9999 [2026-10-20 18:45:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9676 [2026-10-20 18:45:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9636 [2026-10-20 18:45:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7653 [2026-10-20 18:45:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2273 [2026-10-20 18:45:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4211 [2026-10-20 18:45:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9067 [2026-10-20 18:45:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5980 [2026-10-20 18:45:36][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,558 | MFU: 1.8% [2026-10-20 18:45:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9235 [2026-10-20 18:45:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8342 [2026-10-20 18:45:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9804 [2026-10-20 18:45:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2110 [2026-10-20 18:45:40][INFO][training.pretrain_engine] [Ph2|Step 603] loss=5.6984 | ppl=298.39 | aux=0.02509 | lr=3.00e-04 | gnorm=0.762 | tok/s=29,242 | MFU=7.0% | step=143436ms | tokens=0.640B | mem=4.2/32.7GB | cost=$316.90/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 11:10 (136.4h) | Phase3: SKIPPED | SFT: Oct 26 15:55 (141.2h) | DPO (Final): Oct 26 16:29 (141.7h) [2026-10-20 18:45:40][INFO][training.pretrain_engine] Starting global step 604 (accumulating 32 micro-batches)... [2026-10-20 18:45:40][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 18:45:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6604 [2026-10-20 18:45:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5636 [2026-10-20 18:45:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5636 [2026-10-20 18:45:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8395 [2026-10-20 18:46:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7545 [2026-10-20 18:46:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7847 [2026-10-20 18:46:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8509 [2026-10-20 18:46:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8682 [2026-10-20 18:46:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7935 [2026-10-20 18:46:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6743 [2026-10-20 18:46:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5875 [2026-10-20 18:46:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3432 [2026-10-20 18:46:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6687 [2026-10-20 18:46:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4428 [2026-10-20 18:46:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4621 [2026-10-20 18:46:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7781 [2026-10-20 18:47:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6995 [2026-10-20 18:47:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9064 [2026-10-20 18:47:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6875 [2026-10-20 18:47:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4070 [2026-10-20 18:47:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8366 [2026-10-20 18:47:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1610 [2026-10-20 18:47:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9531 [2026-10-20 18:47:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7622 [2026-10-20 18:47:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7320 [2026-10-20 18:47:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5024 [2026-10-20 18:47:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5862 [2026-10-20 18:47:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9189 [2026-10-20 18:48:00][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,507 | MFU: 1.8% [2026-10-20 18:48:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5697 [2026-10-20 18:48:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9249 [2026-10-20 18:48:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4179 [2026-10-20 18:48:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8385 [2026-10-20 18:48:05][INFO][training.pretrain_engine] [Ph2|Step 604] loss=5.7985 | ppl=329.80 | aux=0.02509 | lr=3.00e-04 | gnorm=0.805 | tok/s=29,239 | MFU=7.0% | step=143447ms | tokens=0.641B | mem=4.2/32.7GB | cost=$317.43/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 11:13 (136.4h) | Phase3: SKIPPED | SFT: Oct 26 15:58 (141.2h) | DPO (Final): Oct 26 16:32 (141.7h) [2026-10-20 18:48:05][INFO][training.pretrain_engine] Starting global step 605 (accumulating 32 micro-batches)... [2026-10-20 18:48:05][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 18:48:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7966 [2026-10-20 18:48:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5318 [2026-10-20 18:48:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8670 [2026-10-20 18:48:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4696 [2026-10-20 18:48:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4753 [2026-10-20 18:48:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6454 [2026-10-20 18:48:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6581 [2026-10-20 18:48:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5400 [2026-10-20 18:48:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0531 [2026-10-20 18:48:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5707 [2026-10-20 18:48:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0606 [2026-10-20 18:48:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4115 [2026-10-20 18:49:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7979 [2026-10-20 18:49:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7133 [2026-10-20 18:49:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0273 [2026-10-20 18:49:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7874 [2026-10-20 18:49:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2766 [2026-10-20 18:49:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8668 [2026-10-20 18:49:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0187 [2026-10-20 18:49:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9755 [2026-10-20 18:49:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1001 [2026-10-20 18:49:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1929 [2026-10-20 18:49:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3881 [2026-10-20 18:49:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1479 [2026-10-20 18:50:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4895 [2026-10-20 18:50:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2263 [2026-10-20 18:50:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1954 [2026-10-20 18:50:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8861 [2026-10-20 18:50:24][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,526 | MFU: 1.8% [2026-10-20 18:50:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7461 [2026-10-20 18:50:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9800 [2026-10-20 18:50:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9822 [2026-10-20 18:50:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5810 [2026-10-20 18:50:29][INFO][training.pretrain_engine] [Ph2|Step 605] loss=5.6615 | ppl=287.58 | aux=0.02508 | lr=3.00e-04 | gnorm=0.855 | tok/s=29,238 | MFU=7.0% | step=143455ms | tokens=0.642B | mem=4.2/32.7GB | cost=$317.96/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 11:15 (136.4h) | Phase3: SKIPPED | SFT: Oct 26 16:00 (141.2h) | DPO (Final): Oct 26 16:34 (141.7h) [2026-10-20 18:50:32][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step605 | Tokens: 0.642B | Spent: $317.97 | Remaining: $82.03 [2026-10-20 18:50:32][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step605 [2026-10-20 18:50:32][INFO][training.pretrain_engine] Starting global step 606 (accumulating 32 micro-batches)... [2026-10-20 18:50:32][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 18:50:32][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 18:50:32][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 18:50:32][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 18:50:32][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step605 to Phase-Technologies/forge-500m-pretrain (path: phase2_step605)... [2026-10-20 18:50:32][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 18:50:43][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 18:50:43][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step605 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 18:50:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9603 [2026-10-20 18:50:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1681 [2026-10-20 18:50:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7089 [2026-10-20 18:50:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5754 [2026-10-20 18:51:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3882 [2026-10-20 18:51:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5345 [2026-10-20 18:51:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5054 [2026-10-20 18:51:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6420 [2026-10-20 18:51:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6500 [2026-10-20 18:51:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1655 [2026-10-20 18:51:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.0387 [2026-10-20 18:51:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8455 [2026-10-20 18:51:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7915 [2026-10-20 18:51:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3963 [2026-10-20 18:51:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6080 [2026-10-20 18:51:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0915 [2026-10-20 18:51:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7760 [2026-10-20 18:51:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6141 [2026-10-20 18:51:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8459 [2026-10-20 18:52:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7956 [2026-10-20 18:52:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2841 [2026-10-20 18:52:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9695 [2026-10-20 18:52:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5450 [2026-10-20 18:52:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5579 [2026-10-20 18:52:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0453 [2026-10-20 18:52:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9007 [2026-10-20 18:52:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0306 [2026-10-20 18:52:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3563 [2026-10-20 18:52:51][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,524 | MFU: 1.8% [2026-10-20 18:52:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6293 [2026-10-20 18:52:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2537 [2026-10-20 18:52:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7834 [2026-10-20 18:52:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8658 [2026-10-20 18:52:56][INFO][training.pretrain_engine] [Ph2|Step 606] loss=5.6393 | ppl=281.28 | aux=0.02509 | lr=3.00e-04 | gnorm=0.801 | tok/s=29,238 | MFU=7.0% | step=143455ms | tokens=0.643B | mem=4.2/32.7GB | cost=$318.50/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 11:17 (136.4h) | Phase3: SKIPPED | SFT: Oct 26 16:02 (141.2h) | DPO (Final): Oct 26 16:36 (141.7h) [2026-10-20 18:52:56][INFO][training.pretrain_engine] Starting global step 607 (accumulating 32 micro-batches)... [2026-10-20 18:52:56][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 18:53:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8195 [2026-10-20 18:53:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5441 [2026-10-20 18:53:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8077 [2026-10-20 18:53:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7532 [2026-10-20 18:53:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2250 [2026-10-20 18:53:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7782 [2026-10-20 18:53:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7341 [2026-10-20 18:53:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9455 [2026-10-20 18:53:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6500 [2026-10-20 18:53:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5018 [2026-10-20 18:53:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6378 [2026-10-20 18:53:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5898 [2026-10-20 18:54:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0691 [2026-10-20 18:54:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5383 [2026-10-20 18:54:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5546 [2026-10-20 18:54:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5432 [2026-10-20 18:54:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8794 [2026-10-20 18:54:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0319 [2026-10-20 18:54:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1632 [2026-10-20 18:54:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6182 [2026-10-20 18:54:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5779 [2026-10-20 18:54:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8721 [2026-10-20 18:54:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4640 [2026-10-20 18:54:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5889 [2026-10-20 18:55:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5320 [2026-10-20 18:55:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5470 [2026-10-20 18:55:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8029 [2026-10-20 18:55:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5935 [2026-10-20 18:55:15][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,565 | MFU: 1.8% [2026-10-20 18:55:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7342 [2026-10-20 18:55:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5269 [2026-10-20 18:55:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0095 [2026-10-20 18:55:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6575 [2026-10-20 18:55:20][INFO][training.pretrain_engine] [Ph2|Step 607] loss=5.6645 | ppl=288.44 | aux=0.02509 | lr=3.00e-04 | gnorm=0.738 | tok/s=29,237 | MFU=7.0% | step=143460ms | tokens=0.644B | mem=4.2/32.7GB | cost=$319.02/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 11:19 (136.4h) | Phase3: SKIPPED | SFT: Oct 26 16:04 (141.1h) | DPO (Final): Oct 26 16:38 (141.7h) [2026-10-20 18:55:20][INFO][training.pretrain_engine] Starting global step 608 (accumulating 32 micro-batches)... [2026-10-20 18:55:20][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 18:55:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0470 [2026-10-20 18:55:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0488 [2026-10-20 18:55:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6829 [2026-10-20 18:55:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6295 [2026-10-20 18:55:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5361 [2026-10-20 18:55:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5960 [2026-10-20 18:55:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4325 [2026-10-20 18:55:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0029 [2026-10-20 18:56:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6193 [2026-10-20 18:56:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1017 [2026-10-20 18:56:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3908 [2026-10-20 18:56:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7530 [2026-10-20 18:56:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6689 [2026-10-20 18:56:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5249 [2026-10-20 18:56:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7876 [2026-10-20 18:56:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4372 [2026-10-20 18:56:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2118 [2026-10-20 18:56:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1319 [2026-10-20 18:56:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6727 [2026-10-20 18:56:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7673 [2026-10-20 18:57:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2831 [2026-10-20 18:57:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0525 [2026-10-20 18:57:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6000 [2026-10-20 18:57:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7073 [2026-10-20 18:57:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1642 [2026-10-20 18:57:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5603 [2026-10-20 18:57:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0091 [2026-10-20 18:57:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8445 [2026-10-20 18:57:39][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,522 | MFU: 1.8% [2026-10-20 18:57:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7973 [2026-10-20 18:57:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9138 [2026-10-20 18:57:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4215 [2026-10-20 18:57:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1115 [2026-10-20 18:57:44][INFO][training.pretrain_engine] [Ph2|Step 608] loss=5.5373 | ppl=253.98 | aux=0.02508 | lr=3.00e-04 | gnorm=0.625 | tok/s=29,235 | MFU=7.0% | step=143467ms | tokens=0.645B | mem=4.2/32.7GB | cost=$319.55/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 11:21 (136.4h) | Phase3: SKIPPED | SFT: Oct 26 16:06 (141.1h) | DPO (Final): Oct 26 16:40 (141.7h) [2026-10-20 18:57:44][INFO][training.pretrain_engine] Starting global step 609 (accumulating 32 micro-batches)... [2026-10-20 18:57:44][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 18:58:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4455 [2026-10-20 18:58:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0642 [2026-10-20 18:58:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2041 [2026-10-20 18:58:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1158 [2026-10-20 18:58:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3851 [2026-10-20 18:58:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3849 [2026-10-20 18:58:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4310 [2026-10-20 18:58:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4216 [2026-10-20 18:58:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5131 [2026-10-20 18:58:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4200 [2026-10-20 18:58:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8819 [2026-10-20 18:58:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7741 [2026-10-20 18:58:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3133 [2026-10-20 18:58:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9481 [2026-10-20 18:58:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9135 [2026-10-20 18:58:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8508 [2026-10-20 18:59:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7158 [2026-10-20 18:59:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6021 [2026-10-20 18:59:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8870 [2026-10-20 18:59:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0566 [2026-10-20 18:59:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6537 [2026-10-20 18:59:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6112 [2026-10-20 18:59:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6587 [2026-10-20 18:59:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6208 [2026-10-20 18:59:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7438 [2026-10-20 18:59:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9331 [2026-10-20 18:59:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3592 [2026-10-20 18:59:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0253 [2026-10-20 19:00:03][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,533 | MFU: 1.8% [2026-10-20 19:00:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6744 [2026-10-20 19:00:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8106 [2026-10-20 19:00:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5870 [2026-10-20 19:00:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8309 [2026-10-20 19:00:07][INFO][training.pretrain_engine] [Ph2|Step 609] loss=5.6817 | ppl=293.45 | aux=0.02507 | lr=3.00e-04 | gnorm=0.645 | tok/s=29,233 | MFU=7.0% | step=143476ms | tokens=0.646B | mem=4.2/32.7GB | cost=$320.07/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 11:23 (136.4h) | Phase3: SKIPPED | SFT: Oct 26 16:08 (141.1h) | DPO (Final): Oct 26 16:42 (141.7h) [2026-10-20 19:00:07][INFO][training.pretrain_engine] Starting global step 610 (accumulating 32 micro-batches)... [2026-10-20 19:00:07][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 19:00:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3768 [2026-10-20 19:00:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2876 [2026-10-20 19:00:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3271 [2026-10-20 19:00:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0059 [2026-10-20 19:00:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7474 [2026-10-20 19:00:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4122 [2026-10-20 19:00:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8261 [2026-10-20 19:00:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9618 [2026-10-20 19:01:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7070 [2026-10-20 19:01:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1228 [2026-10-20 19:01:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5902 [2026-10-20 19:01:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5683 [2026-10-20 19:01:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6605 [2026-10-20 19:01:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8996 [2026-10-20 19:01:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7038 [2026-10-20 19:01:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5529 [2026-10-20 19:01:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4558 [2026-10-20 19:01:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8861 [2026-10-20 19:01:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9368 [2026-10-20 19:01:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1204 [2026-10-20 19:01:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7664 [2026-10-20 19:01:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8171 [2026-10-20 19:01:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9674 [2026-10-20 19:01:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7685 [2026-10-20 19:02:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6480 [2026-10-20 19:02:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5824 [2026-10-20 19:02:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5340 [2026-10-20 19:02:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6520 [2026-10-20 19:02:26][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,544 | MFU: 1.8% [2026-10-20 19:02:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5887 [2026-10-20 19:02:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6990 [2026-10-20 19:02:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6244 [2026-10-20 19:02:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2076 [2026-10-20 19:02:31][INFO][training.pretrain_engine] [Ph2|Step 610] loss=5.6659 | ppl=288.86 | aux=0.02508 | lr=3.00e-04 | gnorm=0.668 | tok/s=29,232 | MFU=7.0% | step=143483ms | tokens=0.647B | mem=4.2/32.7GB | cost=$320.60/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 11:25 (136.4h) | Phase3: SKIPPED | SFT: Oct 26 16:10 (141.1h) | DPO (Final): Oct 26 16:45 (141.7h) [2026-10-20 19:02:31][INFO][training.pretrain_engine] Starting global step 611 (accumulating 32 micro-batches)... [2026-10-20 19:02:31][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 19:02:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7720 [2026-10-20 19:02:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6240 [2026-10-20 19:02:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5856 [2026-10-20 19:02:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5586 [2026-10-20 19:03:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2913 [2026-10-20 19:03:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4472 [2026-10-20 19:03:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0813 [2026-10-20 19:03:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2637 [2026-10-20 19:03:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3587 [2026-10-20 19:03:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6839 [2026-10-20 19:03:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2378 [2026-10-20 19:03:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3854 [2026-10-20 19:03:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9903 [2026-10-20 19:03:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8093 [2026-10-20 19:03:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9804 [2026-10-20 19:03:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1021 [2026-10-20 19:04:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2549 [2026-10-20 19:04:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2822 [2026-10-20 19:04:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2442 [2026-10-20 19:04:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5671 [2026-10-20 19:04:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3616 [2026-10-20 19:04:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8008 [2026-10-20 19:04:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1190 [2026-10-20 19:04:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1572 [2026-10-20 19:04:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1540 [2026-10-20 19:04:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3249 [2026-10-20 19:04:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3223 [2026-10-20 19:04:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1835 [2026-10-20 19:04:50][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,545 | MFU: 1.8% [2026-10-20 19:04:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7943 [2026-10-20 19:04:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5722 [2026-10-20 19:04:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4171 [2026-10-20 19:04:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9512 [2026-10-20 19:04:55][INFO][training.pretrain_engine] [Ph2|Step 611] loss=5.5895 | ppl=267.61 | aux=0.02510 | lr=3.00e-04 | gnorm=0.598 | tok/s=29,231 | MFU=7.0% | step=143488ms | tokens=0.648B | mem=4.2/32.7GB | cost=$321.12/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 11:27 (136.4h) | Phase3: SKIPPED | SFT: Oct 26 16:13 (141.1h) | DPO (Final): Oct 26 16:47 (141.7h) [2026-10-20 19:04:55][INFO][training.pretrain_engine] Starting global step 612 (accumulating 32 micro-batches)... [2026-10-20 19:04:55][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 19:05:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3979 [2026-10-20 19:05:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5649 [2026-10-20 19:05:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0206 [2026-10-20 19:05:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4350 [2026-10-20 19:05:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7134 [2026-10-20 19:05:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2621 [2026-10-20 19:05:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6870 [2026-10-20 19:05:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7604 [2026-10-20 19:05:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7287 [2026-10-20 19:05:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7418 [2026-10-20 19:05:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7769 [2026-10-20 19:05:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8622 [2026-10-20 19:06:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5381 [2026-10-20 19:06:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5038 [2026-10-20 19:06:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8727 [2026-10-20 19:06:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4403 [2026-10-20 19:06:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2474 [2026-10-20 19:06:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5974 [2026-10-20 19:06:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3402 [2026-10-20 19:06:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3569 [2026-10-20 19:06:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6688 [2026-10-20 19:06:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7271 [2026-10-20 19:06:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8117 [2026-10-20 19:06:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0837 [2026-10-20 19:07:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3699 [2026-10-20 19:07:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4174 [2026-10-20 19:07:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5101 [2026-10-20 19:07:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6237 [2026-10-20 19:07:13][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,562 | MFU: 1.8% [2026-10-20 19:07:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0140 [2026-10-20 19:07:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7379 [2026-10-20 19:07:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0175 [2026-10-20 19:07:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7690 [2026-10-20 19:07:18][INFO][training.pretrain_engine] [Ph2|Step 612] loss=5.7116 | ppl=302.36 | aux=0.02508 | lr=3.00e-04 | gnorm=0.574 | tok/s=29,230 | MFU=7.0% | step=143492ms | tokens=0.649B | mem=4.2/32.7GB | cost=$321.65/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 11:29 (136.4h) | Phase3: SKIPPED | SFT: Oct 26 16:15 (141.1h) | DPO (Final): Oct 26 16:49 (141.7h) [2026-10-20 19:07:18][INFO][training.pretrain_engine] Starting global step 613 (accumulating 32 micro-batches)... [2026-10-20 19:07:18][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 19:07:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4061 [2026-10-20 19:07:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2257 [2026-10-20 19:07:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8043 [2026-10-20 19:07:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4435 [2026-10-20 19:07:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3400 [2026-10-20 19:07:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6056 [2026-10-20 19:07:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9826 [2026-10-20 19:07:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4638 [2026-10-20 19:08:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5007 [2026-10-20 19:08:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9783 [2026-10-20 19:08:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7750 [2026-10-20 19:08:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8561 [2026-10-20 19:08:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6478 [2026-10-20 19:08:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5849 [2026-10-20 19:08:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2805 [2026-10-20 19:08:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3418 [2026-10-20 19:08:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1918 [2026-10-20 19:08:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8107 [2026-10-20 19:08:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5022 [2026-10-20 19:08:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8715 [2026-10-20 19:09:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4376 [2026-10-20 19:09:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1170 [2026-10-20 19:09:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4640 [2026-10-20 19:09:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1934 [2026-10-20 19:09:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9394 [2026-10-20 19:09:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9802 [2026-10-20 19:09:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7808 [2026-10-20 19:09:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1425 [2026-10-20 19:09:37][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,557 | MFU: 1.8% [2026-10-20 19:09:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0585 [2026-10-20 19:09:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6482 [2026-10-20 19:09:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6597 [2026-10-20 19:09:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8991 [2026-10-20 19:09:42][INFO][training.pretrain_engine] [Ph2|Step 613] loss=5.6873 | ppl=295.09 | aux=0.02508 | lr=3.00e-04 | gnorm=0.605 | tok/s=29,229 | MFU=7.0% | step=143496ms | tokens=0.650B | mem=4.2/32.7GB | cost=$322.17/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 11:32 (136.4h) | Phase3: SKIPPED | SFT: Oct 26 16:17 (141.1h) | DPO (Final): Oct 26 16:51 (141.7h) [2026-10-20 19:09:42][INFO][training.pretrain_engine] Starting global step 614 (accumulating 32 micro-batches)... [2026-10-20 19:09:42][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 19:10:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0909 [2026-10-20 19:10:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9266 [2026-10-20 19:10:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7405 [2026-10-20 19:10:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0871 [2026-10-20 19:10:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5964 [2026-10-20 19:10:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9047 [2026-10-20 19:10:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1993 [2026-10-20 19:10:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5170 [2026-10-20 19:10:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4611 [2026-10-20 19:10:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0327 [2026-10-20 19:10:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7635 [2026-10-20 19:10:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9188 [2026-10-20 19:10:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8439 [2026-10-20 19:10:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5494 [2026-10-20 19:10:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5127 [2026-10-20 19:10:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8232 [2026-10-20 19:11:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5673 [2026-10-20 19:11:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7645 [2026-10-20 19:11:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7671 [2026-10-20 19:11:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7516 [2026-10-20 19:11:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8146 [2026-10-20 19:11:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5083 [2026-10-20 19:11:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4194 [2026-10-20 19:11:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5014 [2026-10-20 19:11:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8023 [2026-10-20 19:11:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8103 [2026-10-20 19:11:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0214 [2026-10-20 19:11:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8009 [2026-10-20 19:12:01][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,551 | MFU: 1.8% [2026-10-20 19:12:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3439 [2026-10-20 19:12:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6481 [2026-10-20 19:12:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7566 [2026-10-20 19:12:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1220 [2026-10-20 19:12:06][INFO][training.pretrain_engine] [Ph2|Step 614] loss=5.6240 | ppl=277.00 | aux=0.02508 | lr=3.00e-04 | gnorm=0.551 | tok/s=29,228 | MFU=7.0% | step=143502ms | tokens=0.651B | mem=4.2/32.7GB | cost=$322.70/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 11:34 (136.4h) | Phase3: SKIPPED | SFT: Oct 26 16:19 (141.1h) | DPO (Final): Oct 26 16:53 (141.7h) [2026-10-20 19:12:09][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step614 | Tokens: 0.651B | Spent: $322.71 | Remaining: $77.29 [2026-10-20 19:12:09][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step614 [2026-10-20 19:12:09][INFO][training.pretrain_engine] Starting global step 615 (accumulating 32 micro-batches)... [2026-10-20 19:12:09][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 19:12:09][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 19:12:09][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 19:12:09][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 19:12:09][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step614 to Phase-Technologies/forge-500m-pretrain (path: phase2_step614)... [2026-10-20 19:12:10][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 19:12:19][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 19:12:19][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step614 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 19:12:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6262 [2026-10-20 19:12:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3439 [2026-10-20 19:12:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0901 [2026-10-20 19:12:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3994 [2026-10-20 19:12:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1237 [2026-10-20 19:12:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7708 [2026-10-20 19:12:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0483 [2026-10-20 19:12:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3005 [2026-10-20 19:12:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6819 [2026-10-20 19:12:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4339 [2026-10-20 19:12:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7842 [2026-10-20 19:13:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2341 [2026-10-20 19:13:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4017 [2026-10-20 19:13:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2603 [2026-10-20 19:13:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6143 [2026-10-20 19:13:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6573 [2026-10-20 19:13:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6781 [2026-10-20 19:13:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6695 [2026-10-20 19:13:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5369 [2026-10-20 19:13:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9453 [2026-10-20 19:13:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8382 [2026-10-20 19:13:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7545 [2026-10-20 19:13:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4101 [2026-10-20 19:13:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9027 [2026-10-20 19:14:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4940 [2026-10-20 19:14:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8488 [2026-10-20 19:14:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9322 [2026-10-20 19:14:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6971 [2026-10-20 19:14:29][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,498 | MFU: 1.8% [2026-10-20 19:14:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6386 [2026-10-20 19:14:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6177 [2026-10-20 19:14:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6988 [2026-10-20 19:14:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4456 [2026-10-20 19:14:34][INFO][training.pretrain_engine] [Ph2|Step 615] loss=5.6434 | ppl=282.43 | aux=0.02508 | lr=3.00e-04 | gnorm=0.711 | tok/s=29,226 | MFU=7.0% | step=143513ms | tokens=0.652B | mem=4.2/32.7GB | cost=$323.24/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 11:36 (136.4h) | Phase3: SKIPPED | SFT: Oct 26 16:21 (141.1h) | DPO (Final): Oct 26 16:55 (141.7h) [2026-10-20 19:14:34][INFO][training.pretrain_engine] Starting global step 616 (accumulating 32 micro-batches)... [2026-10-20 19:14:34][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 19:14:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9339 [2026-10-20 19:14:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8991 [2026-10-20 19:14:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6174 [2026-10-20 19:14:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9200 [2026-10-20 19:15:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6695 [2026-10-20 19:15:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5871 [2026-10-20 19:15:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6068 [2026-10-20 19:15:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6368 [2026-10-20 19:15:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7352 [2026-10-20 19:15:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1041 [2026-10-20 19:15:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1899 [2026-10-20 19:15:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6804 [2026-10-20 19:15:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8400 [2026-10-20 19:15:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4204 [2026-10-20 19:15:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6352 [2026-10-20 19:15:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5813 [2026-10-20 19:16:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4768 [2026-10-20 19:16:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3495 [2026-10-20 19:16:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2855 [2026-10-20 19:16:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2698 [2026-10-20 19:16:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5021 [2026-10-20 19:16:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9551 [2026-10-20 19:16:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5691 [2026-10-20 19:16:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1165 [2026-10-20 19:16:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6411 [2026-10-20 19:16:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1507 [2026-10-20 19:16:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5632 [2026-10-20 19:16:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8828 [2026-10-20 19:16:53][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,512 | MFU: 1.8% [2026-10-20 19:16:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5338 [2026-10-20 19:16:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4721 [2026-10-20 19:16:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3397 [2026-10-20 19:16:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4187 [2026-10-20 19:16:58][INFO][training.pretrain_engine] [Ph2|Step 616] loss=5.6274 | ppl=277.93 | aux=0.02509 | lr=3.00e-04 | gnorm=0.672 | tok/s=29,225 | MFU=7.0% | step=143518ms | tokens=0.653B | mem=4.2/32.7GB | cost=$323.77/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 11:38 (136.4h) | Phase3: SKIPPED | SFT: Oct 26 16:23 (141.1h) | DPO (Final): Oct 26 16:58 (141.7h) [2026-10-20 19:16:58][INFO][training.pretrain_engine] Starting global step 617 (accumulating 32 micro-batches)... [2026-10-20 19:16:58][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 19:17:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4236 [2026-10-20 19:17:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8126 [2026-10-20 19:17:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8436 [2026-10-20 19:17:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0742 [2026-10-20 19:17:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0276 [2026-10-20 19:17:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9018 [2026-10-20 19:17:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5169 [2026-10-20 19:17:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3783 [2026-10-20 19:17:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5061 [2026-10-20 19:17:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7557 [2026-10-20 19:17:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4615 [2026-10-20 19:17:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5137 [2026-10-20 19:18:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1894 [2026-10-20 19:18:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0653 [2026-10-20 19:18:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1443 [2026-10-20 19:18:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8614 [2026-10-20 19:18:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7093 [2026-10-20 19:18:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1252 [2026-10-20 19:18:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8472 [2026-10-20 19:18:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8560 [2026-10-20 19:18:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5915 [2026-10-20 19:18:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0144 [2026-10-20 19:18:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1951 [2026-10-20 19:18:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8200 [2026-10-20 19:19:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7825 [2026-10-20 19:19:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1579 [2026-10-20 19:19:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2723 [2026-10-20 19:19:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8610 [2026-10-20 19:19:17][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,552 | MFU: 1.8% [2026-10-20 19:19:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5689 [2026-10-20 19:19:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8763 [2026-10-20 19:19:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8737 [2026-10-20 19:19:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8296 [2026-10-20 19:19:21][INFO][training.pretrain_engine] [Ph2|Step 617] loss=5.6827 | ppl=293.75 | aux=0.02508 | lr=3.00e-04 | gnorm=0.637 | tok/s=29,224 | MFU=7.0% | step=143521ms | tokens=0.654B | mem=4.2/32.7GB | cost=$324.29/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 11:40 (136.4h) | Phase3: SKIPPED | SFT: Oct 26 16:25 (141.1h) | DPO (Final): Oct 26 17:00 (141.7h) [2026-10-20 19:19:21][INFO][training.pretrain_engine] Starting global step 618 (accumulating 32 micro-batches)... [2026-10-20 19:19:21][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 19:19:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7079 [2026-10-20 19:19:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5087 [2026-10-20 19:19:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9562 [2026-10-20 19:19:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7348 [2026-10-20 19:19:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7629 [2026-10-20 19:19:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8848 [2026-10-20 19:19:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6817 [2026-10-20 19:19:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5096 [2026-10-20 19:20:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6711 [2026-10-20 19:20:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8725 [2026-10-20 19:20:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6907 [2026-10-20 19:20:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6288 [2026-10-20 19:20:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9986 [2026-10-20 19:20:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5718 [2026-10-20 19:20:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0671 [2026-10-20 19:20:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7895 [2026-10-20 19:20:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3483 [2026-10-20 19:20:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7175 [2026-10-20 19:20:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8321 [2026-10-20 19:20:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4763 [2026-10-20 19:21:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9907 [2026-10-20 19:21:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3561 [2026-10-20 19:21:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4528 [2026-10-20 19:21:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2261 [2026-10-20 19:21:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9589 [2026-10-20 19:21:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7883 [2026-10-20 19:21:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4370 [2026-10-20 19:21:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2747 [2026-10-20 19:21:40][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,567 | MFU: 1.8% [2026-10-20 19:21:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6960 [2026-10-20 19:21:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9135 [2026-10-20 19:21:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6899 [2026-10-20 19:21:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6190 [2026-10-20 19:21:44][INFO][training.pretrain_engine] [Ph2|Step 618] loss=5.7042 | ppl=300.12 | aux=0.02509 | lr=3.00e-04 | gnorm=0.758 | tok/s=29,225 | MFU=7.0% | step=143519ms | tokens=0.655B | mem=4.2/32.7GB | cost=$324.81/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 11:42 (136.3h) | Phase3: SKIPPED | SFT: Oct 26 16:27 (141.1h) | DPO (Final): Oct 26 17:01 (141.7h) [2026-10-20 19:21:44][INFO][training.pretrain_engine] Starting global step 619 (accumulating 32 micro-batches)... [2026-10-20 19:21:44][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 19:22:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7519 [2026-10-20 19:22:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8060 [2026-10-20 19:22:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9525 [2026-10-20 19:22:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9171 [2026-10-20 19:22:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6208 [2026-10-20 19:22:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8299 [2026-10-20 19:22:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1360 [2026-10-20 19:22:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4965 [2026-10-20 19:22:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8826 [2026-10-20 19:22:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6570 [2026-10-20 19:22:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6670 [2026-10-20 19:22:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8801 [2026-10-20 19:22:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3967 [2026-10-20 19:22:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0614 [2026-10-20 19:22:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8001 [2026-10-20 19:22:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8496 [2026-10-20 19:23:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5420 [2026-10-20 19:23:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3650 [2026-10-20 19:23:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4927 [2026-10-20 19:23:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4390 [2026-10-20 19:23:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5685 [2026-10-20 19:23:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7701 [2026-10-20 19:23:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7394 [2026-10-20 19:23:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6471 [2026-10-20 19:23:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5264 [2026-10-20 19:23:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0610 [2026-10-20 19:23:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0735 [2026-10-20 19:23:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6649 [2026-10-20 19:24:03][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,573 | MFU: 1.8% [2026-10-20 19:24:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7632 [2026-10-20 19:24:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6267 [2026-10-20 19:24:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8031 [2026-10-20 19:24:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9298 [2026-10-20 19:24:08][INFO][training.pretrain_engine] [Ph2|Step 619] loss=5.6387 | ppl=281.11 | aux=0.02509 | lr=3.00e-04 | gnorm=0.926 | tok/s=29,225 | MFU=7.0% | step=143518ms | tokens=0.656B | mem=4.2/32.7GB | cost=$325.34/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 11:44 (136.3h) | Phase3: SKIPPED | SFT: Oct 26 16:29 (141.1h) | DPO (Final): Oct 26 17:03 (141.7h) [2026-10-20 19:24:08][INFO][training.pretrain_engine] Starting global step 620 (accumulating 32 micro-batches)... [2026-10-20 19:24:08][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 19:24:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7750 [2026-10-20 19:24:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6397 [2026-10-20 19:24:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9136 [2026-10-20 19:24:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9763 [2026-10-20 19:24:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8761 [2026-10-20 19:24:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9256 [2026-10-20 19:24:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0405 [2026-10-20 19:24:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.5389 [2026-10-20 19:25:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7539 [2026-10-20 19:25:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2713 [2026-10-20 19:25:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5576 [2026-10-20 19:25:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3824 [2026-10-20 19:25:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9156 [2026-10-20 19:25:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4492 [2026-10-20 19:25:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9660 [2026-10-20 19:25:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6598 [2026-10-20 19:25:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7160 [2026-10-20 19:25:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8708 [2026-10-20 19:25:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0532 [2026-10-20 19:25:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8666 [2026-10-20 19:25:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4091 [2026-10-20 19:25:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9744 [2026-10-20 19:25:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8095 [2026-10-20 19:25:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1243 [2026-10-20 19:26:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8910 [2026-10-20 19:26:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7588 [2026-10-20 19:26:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3924 [2026-10-20 19:26:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6198 [2026-10-20 19:26:26][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,568 | MFU: 1.8% [2026-10-20 19:26:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2066 [2026-10-20 19:26:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4706 [2026-10-20 19:26:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1572 [2026-10-20 19:26:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2857 [2026-10-20 19:26:31][INFO][training.pretrain_engine] [Ph2|Step 620] loss=5.5819 | ppl=265.58 | aux=0.02510 | lr=3.00e-04 | gnorm=1.242 | tok/s=29,225 | MFU=7.0% | step=143520ms | tokens=0.657B | mem=4.2/32.7GB | cost=$325.86/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 11:46 (136.3h) | Phase3: SKIPPED | SFT: Oct 26 16:31 (141.1h) | DPO (Final): Oct 26 17:05 (141.6h) [2026-10-20 19:26:31][INFO][training.pretrain_engine] Starting global step 621 (accumulating 32 micro-batches)... [2026-10-20 19:26:31][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 19:26:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6526 [2026-10-20 19:26:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8085 [2026-10-20 19:26:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2020 [2026-10-20 19:26:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6897 [2026-10-20 19:27:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2017 [2026-10-20 19:27:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8624 [2026-10-20 19:27:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4713 [2026-10-20 19:27:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4043 [2026-10-20 19:27:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3709 [2026-10-20 19:27:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9167 [2026-10-20 19:27:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7709 [2026-10-20 19:27:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0293 [2026-10-20 19:27:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9567 [2026-10-20 19:27:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9132 [2026-10-20 19:27:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1545 [2026-10-20 19:27:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1014 [2026-10-20 19:28:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6991 [2026-10-20 19:28:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1638 [2026-10-20 19:28:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6253 [2026-10-20 19:28:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3267 [2026-10-20 19:28:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5870 [2026-10-20 19:28:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5865 [2026-10-20 19:28:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5467 [2026-10-20 19:28:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3768 [2026-10-20 19:28:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7435 [2026-10-20 19:28:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7414 [2026-10-20 19:28:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1505 [2026-10-20 19:28:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6643 [2026-10-20 19:28:49][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,578 | MFU: 1.8% [2026-10-20 19:28:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6438 [2026-10-20 19:28:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4719 [2026-10-20 19:28:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4761 [2026-10-20 19:28:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5020 [2026-10-20 19:28:54][INFO][training.pretrain_engine] [Ph2|Step 621] loss=5.6751 | ppl=291.53 | aux=0.02509 | lr=3.00e-04 | gnorm=0.824 | tok/s=29,225 | MFU=7.0% | step=143516ms | tokens=0.659B | mem=4.2/32.7GB | cost=$326.38/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 11:47 (136.3h) | Phase3: SKIPPED | SFT: Oct 26 16:32 (141.1h) | DPO (Final): Oct 26 17:06 (141.6h) [2026-10-20 19:28:54][INFO][training.pretrain_engine] Starting global step 622 (accumulating 32 micro-batches)... [2026-10-20 19:28:54][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 19:29:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0008 [2026-10-20 19:29:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9793 [2026-10-20 19:29:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4586 [2026-10-20 19:29:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0235 [2026-10-20 19:29:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3215 [2026-10-20 19:29:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6507 [2026-10-20 19:29:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3812 [2026-10-20 19:29:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3704 [2026-10-20 19:29:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7436 [2026-10-20 19:29:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2237 [2026-10-20 19:29:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5274 [2026-10-20 19:29:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6567 [2026-10-20 19:30:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4267 [2026-10-20 19:30:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4257 [2026-10-20 19:30:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0089 [2026-10-20 19:30:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1978 [2026-10-20 19:30:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3291 [2026-10-20 19:30:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8256 [2026-10-20 19:30:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9201 [2026-10-20 19:30:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5594 [2026-10-20 19:30:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3303 [2026-10-20 19:30:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2710 [2026-10-20 19:30:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8151 [2026-10-20 19:30:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3648 [2026-10-20 19:30:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3950 [2026-10-20 19:30:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4249 [2026-10-20 19:30:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0307 [2026-10-20 19:30:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1697 [2026-10-20 19:31:12][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,583 | MFU: 1.8% [2026-10-20 19:31:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1252 [2026-10-20 19:31:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3073 [2026-10-20 19:31:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2845 [2026-10-20 19:31:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1216 [2026-10-20 19:31:17][INFO][training.pretrain_engine] [Ph2|Step 622] loss=5.7131 | ppl=302.81 | aux=0.02510 | lr=3.00e-04 | gnorm=0.879 | tok/s=29,225 | MFU=7.0% | step=143516ms | tokens=0.660B | mem=4.2/32.7GB | cost=$326.91/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 11:49 (136.3h) | Phase3: SKIPPED | SFT: Oct 26 16:34 (141.1h) | DPO (Final): Oct 26 17:08 (141.6h) [2026-10-20 19:31:17][INFO][training.pretrain_engine] Starting global step 623 (accumulating 32 micro-batches)... [2026-10-20 19:31:17][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 19:31:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5326 [2026-10-20 19:31:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2116 [2026-10-20 19:31:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4043 [2026-10-20 19:31:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3998 [2026-10-20 19:31:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3820 [2026-10-20 19:31:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9060 [2026-10-20 19:31:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7084 [2026-10-20 19:31:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6625 [2026-10-20 19:32:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9625 [2026-10-20 19:32:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9135 [2026-10-20 19:32:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2496 [2026-10-20 19:32:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2300 [2026-10-20 19:32:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5207 [2026-10-20 19:32:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8040 [2026-10-20 19:32:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1123 [2026-10-20 19:32:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8258 [2026-10-20 19:32:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0110 [2026-10-20 19:32:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7772 [2026-10-20 19:32:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0679 [2026-10-20 19:32:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7428 [2026-10-20 19:33:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8928 [2026-10-20 19:33:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6212 [2026-10-20 19:33:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7830 [2026-10-20 19:33:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1077 [2026-10-20 19:33:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2883 [2026-10-20 19:33:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4865 [2026-10-20 19:33:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0894 [2026-10-20 19:33:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5926 [2026-10-20 19:33:36][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,563 | MFU: 1.8% [2026-10-20 19:33:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2308 [2026-10-20 19:33:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9843 [2026-10-20 19:33:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9060 [2026-10-20 19:33:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4477 [2026-10-20 19:33:40][INFO][training.pretrain_engine] [Ph2|Step 623] loss=5.7516 | ppl=314.70 | aux=0.02509 | lr=3.00e-04 | gnorm=1.133 | tok/s=29,226 | MFU=7.0% | step=143514ms | tokens=0.661B | mem=4.2/32.7GB | cost=$327.43/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 11:51 (136.3h) | Phase3: SKIPPED | SFT: Oct 26 16:36 (141.0h) | DPO (Final): Oct 26 17:10 (141.6h) [2026-10-20 19:33:44][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step623 | Tokens: 0.661B | Spent: $327.44 | Remaining: $72.56 [2026-10-20 19:33:44][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step623 [2026-10-20 19:33:44][INFO][training.pretrain_engine] Starting global step 624 (accumulating 32 micro-batches)... [2026-10-20 19:33:44][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 19:33:44][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 19:33:44][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 19:33:44][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 19:33:44][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step623 to Phase-Technologies/forge-500m-pretrain (path: phase2_step623)... [2026-10-20 19:33:44][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 19:33:54][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 19:33:54][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step623 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 19:33:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5552 [2026-10-20 19:33:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5089 [2026-10-20 19:33:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3201 [2026-10-20 19:34:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5183 [2026-10-20 19:34:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5297 [2026-10-20 19:34:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7556 [2026-10-20 19:34:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4286 [2026-10-20 19:34:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5921 [2026-10-20 19:34:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4873 [2026-10-20 19:34:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9513 [2026-10-20 19:34:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4828 [2026-10-20 19:34:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2815 [2026-10-20 19:34:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7451 [2026-10-20 19:34:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6069 [2026-10-20 19:34:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9100 [2026-10-20 19:34:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5509 [2026-10-20 19:35:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7245 [2026-10-20 19:35:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3361 [2026-10-20 19:35:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2170 [2026-10-20 19:35:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8522 [2026-10-20 19:35:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5638 [2026-10-20 19:35:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6899 [2026-10-20 19:35:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5296 [2026-10-20 19:35:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9615 [2026-10-20 19:35:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7837 [2026-10-20 19:35:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6762 [2026-10-20 19:35:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5256 [2026-10-20 19:35:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9163 [2026-10-20 19:36:03][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,553 | MFU: 1.8% [2026-10-20 19:36:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7995 [2026-10-20 19:36:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4137 [2026-10-20 19:36:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7955 [2026-10-20 19:36:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4846 [2026-10-20 19:36:07][INFO][training.pretrain_engine] [Ph2|Step 624] loss=5.6139 | ppl=274.22 | aux=0.02508 | lr=3.00e-04 | gnorm=0.961 | tok/s=29,226 | MFU=7.0% | step=143514ms | tokens=0.662B | mem=4.2/32.7GB | cost=$327.97/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 11:52 (136.3h) | Phase3: SKIPPED | SFT: Oct 26 16:38 (141.0h) | DPO (Final): Oct 26 17:12 (141.6h) [2026-10-20 19:36:07][INFO][training.pretrain_engine] Starting global step 625 (accumulating 32 micro-batches)... [2026-10-20 19:36:07][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 19:36:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4261 [2026-10-20 19:36:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1994 [2026-10-20 19:36:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5424 [2026-10-20 19:36:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6673 [2026-10-20 19:36:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1032 [2026-10-20 19:36:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8690 [2026-10-20 19:36:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6274 [2026-10-20 19:36:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5660 [2026-10-20 19:37:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6846 [2026-10-20 19:37:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9810 [2026-10-20 19:37:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7516 [2026-10-20 19:37:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0332 [2026-10-20 19:37:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6418 [2026-10-20 19:37:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5105 [2026-10-20 19:37:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2071 [2026-10-20 19:37:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5601 [2026-10-20 19:37:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5950 [2026-10-20 19:37:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5581 [2026-10-20 19:37:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6144 [2026-10-20 19:37:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2396 [2026-10-20 19:37:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1840 [2026-10-20 19:37:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9837 [2026-10-20 19:37:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8911 [2026-10-20 19:37:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0611 [2026-10-20 19:38:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1838 [2026-10-20 19:38:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1085 [2026-10-20 19:38:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6942 [2026-10-20 19:38:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5450 [2026-10-20 19:38:26][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,554 | MFU: 1.8% [2026-10-20 19:38:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6217 [2026-10-20 19:38:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6467 [2026-10-20 19:38:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6621 [2026-10-20 19:38:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2795 [2026-10-20 19:38:30][INFO][training.pretrain_engine] [Ph2|Step 625] loss=5.5957 | ppl=269.27 | aux=0.02509 | lr=3.00e-04 | gnorm=0.910 | tok/s=29,225 | MFU=7.0% | step=143516ms | tokens=0.663B | mem=4.2/32.7GB | cost=$328.49/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 11:54 (136.3h) | Phase3: SKIPPED | SFT: Oct 26 16:39 (141.0h) | DPO (Final): Oct 26 17:14 (141.6h) [2026-10-20 19:38:30][INFO][training.pretrain_engine] Starting global step 626 (accumulating 32 micro-batches)... [2026-10-20 19:38:30][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 19:38:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1322 [2026-10-20 19:38:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6993 [2026-10-20 19:38:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5816 [2026-10-20 19:38:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0628 [2026-10-20 19:39:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5316 [2026-10-20 19:39:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6568 [2026-10-20 19:39:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7512 [2026-10-20 19:39:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8153 [2026-10-20 19:39:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0245 [2026-10-20 19:39:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4373 [2026-10-20 19:39:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4294 [2026-10-20 19:39:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6578 [2026-10-20 19:39:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8228 [2026-10-20 19:39:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8253 [2026-10-20 19:39:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8358 [2026-10-20 19:39:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7528 [2026-10-20 19:39:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3081 [2026-10-20 19:39:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5704 [2026-10-20 19:40:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5359 [2026-10-20 19:40:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4845 [2026-10-20 19:40:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0612 [2026-10-20 19:40:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4848 [2026-10-20 19:40:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4212 [2026-10-20 19:40:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7383 [2026-10-20 19:40:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0255 [2026-10-20 19:40:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9091 [2026-10-20 19:40:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6064 [2026-10-20 19:40:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4725 [2026-10-20 19:40:49][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,556 | MFU: 1.8% [2026-10-20 19:40:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2173 [2026-10-20 19:40:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6545 [2026-10-20 19:40:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2409 [2026-10-20 19:40:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1297 [2026-10-20 19:40:54][INFO][training.pretrain_engine] [Ph2|Step 626] loss=5.5232 | ppl=250.42 | aux=0.02509 | lr=3.00e-04 | gnorm=0.902 | tok/s=29,226 | MFU=7.0% | step=143514ms | tokens=0.664B | mem=4.2/32.7GB | cost=$329.01/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 11:56 (136.3h) | Phase3: SKIPPED | SFT: Oct 26 16:41 (141.0h) | DPO (Final): Oct 26 17:15 (141.6h) [2026-10-20 19:40:54][INFO][training.pretrain_engine] Starting global step 627 (accumulating 32 micro-batches)... [2026-10-20 19:40:54][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 19:41:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4813 [2026-10-20 19:41:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4148 [2026-10-20 19:41:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4238 [2026-10-20 19:41:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2249 [2026-10-20 19:41:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7917 [2026-10-20 19:41:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0832 [2026-10-20 19:41:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3782 [2026-10-20 19:41:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3062 [2026-10-20 19:41:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9412 [2026-10-20 19:41:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5791 [2026-10-20 19:41:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5255 [2026-10-20 19:41:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7024 [2026-10-20 19:42:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3881 [2026-10-20 19:42:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4171 [2026-10-20 19:42:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8661 [2026-10-20 19:42:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2618 [2026-10-20 19:42:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5027 [2026-10-20 19:42:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1951 [2026-10-20 19:42:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4863 [2026-10-20 19:42:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6878 [2026-10-20 19:42:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7796 [2026-10-20 19:42:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2870 [2026-10-20 19:42:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6364 [2026-10-20 19:42:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6963 [2026-10-20 19:42:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6889 [2026-10-20 19:42:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8339 [2026-10-20 19:42:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6837 [2026-10-20 19:42:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7556 [2026-10-20 19:43:12][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,573 | MFU: 1.8% [2026-10-20 19:43:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3394 [2026-10-20 19:43:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3665 [2026-10-20 19:43:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6397 [2026-10-20 19:43:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3685 [2026-10-20 19:43:17][INFO][training.pretrain_engine] [Ph2|Step 627] loss=5.6567 | ppl=286.20 | aux=0.02508 | lr=3.00e-04 | gnorm=0.863 | tok/s=29,226 | MFU=7.0% | step=143511ms | tokens=0.665B | mem=4.2/32.7GB | cost=$329.54/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 11:58 (136.2h) | Phase3: SKIPPED | SFT: Oct 26 16:43 (141.0h) | DPO (Final): Oct 26 17:17 (141.6h) [2026-10-20 19:43:17][INFO][training.pretrain_engine] Starting global step 628 (accumulating 32 micro-batches)... [2026-10-20 19:43:17][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 19:43:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5769 [2026-10-20 19:43:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7440 [2026-10-20 19:43:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8108 [2026-10-20 19:43:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5270 [2026-10-20 19:43:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5462 [2026-10-20 19:43:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1675 [2026-10-20 19:43:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2211 [2026-10-20 19:43:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2221 [2026-10-20 19:44:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3534 [2026-10-20 19:44:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9507 [2026-10-20 19:44:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0771 [2026-10-20 19:44:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9877 [2026-10-20 19:44:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8838 [2026-10-20 19:44:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1969 [2026-10-20 19:44:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9689 [2026-10-20 19:44:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5411 [2026-10-20 19:44:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7883 [2026-10-20 19:44:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5366 [2026-10-20 19:44:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5707 [2026-10-20 19:44:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8200 [2026-10-20 19:45:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0538 [2026-10-20 19:45:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9219 [2026-10-20 19:45:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7480 [2026-10-20 19:45:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2726 [2026-10-20 19:45:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4673 [2026-10-20 19:45:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1859 [2026-10-20 19:45:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6739 [2026-10-20 19:45:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4692 [2026-10-20 19:45:35][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,579 | MFU: 1.8% [2026-10-20 19:45:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8611 [2026-10-20 19:45:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9898 [2026-10-20 19:45:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3554 [2026-10-20 19:45:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2669 [2026-10-20 19:45:40][INFO][training.pretrain_engine] [Ph2|Step 628] loss=5.6438 | ppl=282.55 | aux=0.02509 | lr=3.00e-04 | gnorm=1.062 | tok/s=29,226 | MFU=7.0% | step=143513ms | tokens=0.666B | mem=4.2/32.7GB | cost=$330.06/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 12:00 (136.2h) | Phase3: SKIPPED | SFT: Oct 26 16:45 (141.0h) | DPO (Final): Oct 26 17:19 (141.6h) [2026-10-20 19:45:40][INFO][training.pretrain_engine] Starting global step 629 (accumulating 32 micro-batches)... [2026-10-20 19:45:40][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 19:45:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4282 [2026-10-20 19:45:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1460 [2026-10-20 19:45:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7315 [2026-10-20 19:45:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5961 [2026-10-20 19:46:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3569 [2026-10-20 19:46:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0185 [2026-10-20 19:46:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5897 [2026-10-20 19:46:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0559 [2026-10-20 19:46:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7497 [2026-10-20 19:46:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7282 [2026-10-20 19:46:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5668 [2026-10-20 19:46:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3173 [2026-10-20 19:46:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9328 [2026-10-20 19:46:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8973 [2026-10-20 19:46:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4901 [2026-10-20 19:46:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2421 [2026-10-20 19:47:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7211 [2026-10-20 19:47:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5358 [2026-10-20 19:47:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5563 [2026-10-20 19:47:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5427 [2026-10-20 19:47:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0970 [2026-10-20 19:47:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1155 [2026-10-20 19:47:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1586 [2026-10-20 19:47:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4542 [2026-10-20 19:47:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0075 [2026-10-20 19:47:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1494 [2026-10-20 19:47:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1438 [2026-10-20 19:47:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7771 [2026-10-20 19:47:59][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,573 | MFU: 1.8% [2026-10-20 19:48:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2488 [2026-10-20 19:48:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9694 [2026-10-20 19:48:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9988 [2026-10-20 19:48:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7914 [2026-10-20 19:48:04][INFO][training.pretrain_engine] [Ph2|Step 629] loss=5.6215 | ppl=276.31 | aux=0.02507 | lr=3.00e-04 | gnorm=1.062 | tok/s=29,226 | MFU=7.0% | step=143512ms | tokens=0.667B | mem=4.2/32.7GB | cost=$330.59/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 12:01 (136.2h) | Phase3: SKIPPED | SFT: Oct 26 16:46 (141.0h) | DPO (Final): Oct 26 17:21 (141.6h) [2026-10-20 19:48:04][INFO][training.pretrain_engine] Starting global step 630 (accumulating 32 micro-batches)... [2026-10-20 19:48:04][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 19:48:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5201 [2026-10-20 19:48:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2625 [2026-10-20 19:48:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9195 [2026-10-20 19:48:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4706 [2026-10-20 19:48:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8508 [2026-10-20 19:48:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1000 [2026-10-20 19:48:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7595 [2026-10-20 19:48:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0769 [2026-10-20 19:48:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5583 [2026-10-20 19:48:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2802 [2026-10-20 19:48:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4258 [2026-10-20 19:48:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6613 [2026-10-20 19:49:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4260 [2026-10-20 19:49:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0014 [2026-10-20 19:49:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2498 [2026-10-20 19:49:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3565 [2026-10-20 19:49:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0120 [2026-10-20 19:49:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8577 [2026-10-20 19:49:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1153 [2026-10-20 19:49:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6927 [2026-10-20 19:49:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7848 [2026-10-20 19:49:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6218 [2026-10-20 19:49:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2687 [2026-10-20 19:49:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5230 [2026-10-20 19:50:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6968 [2026-10-20 19:50:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4050 [2026-10-20 19:50:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8281 [2026-10-20 19:50:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9949 [2026-10-20 19:50:22][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,583 | MFU: 1.8% [2026-10-20 19:50:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8314 [2026-10-20 19:50:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7878 [2026-10-20 19:50:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7494 [2026-10-20 19:50:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0588 [2026-10-20 19:50:27][INFO][training.pretrain_engine] [Ph2|Step 630] loss=5.6440 | ppl=282.59 | aux=0.02508 | lr=3.00e-04 | gnorm=1.016 | tok/s=29,226 | MFU=7.0% | step=143510ms | tokens=0.668B | mem=4.2/32.7GB | cost=$331.11/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 12:03 (136.2h) | Phase3: SKIPPED | SFT: Oct 26 16:48 (141.0h) | DPO (Final): Oct 26 17:22 (141.5h) [2026-10-20 19:50:27][INFO][training.pretrain_engine] Starting global step 631 (accumulating 32 micro-batches)... [2026-10-20 19:50:27][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 19:50:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5762 [2026-10-20 19:50:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4457 [2026-10-20 19:50:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0327 [2026-10-20 19:50:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7088 [2026-10-20 19:51:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5040 [2026-10-20 19:51:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7108 [2026-10-20 19:51:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6261 [2026-10-20 19:51:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5421 [2026-10-20 19:51:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4738 [2026-10-20 19:51:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7674 [2026-10-20 19:51:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4571 [2026-10-20 19:51:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1448 [2026-10-20 19:51:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8141 [2026-10-20 19:51:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8581 [2026-10-20 19:51:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0443 [2026-10-20 19:51:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6283 [2026-10-20 19:51:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6712 [2026-10-20 19:51:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5191 [2026-10-20 19:51:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5426 [2026-10-20 19:51:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9407 [2026-10-20 19:52:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5022 [2026-10-20 19:52:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5636 [2026-10-20 19:52:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4292 [2026-10-20 19:52:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7712 [2026-10-20 19:52:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3341 [2026-10-20 19:52:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7659 [2026-10-20 19:52:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7669 [2026-10-20 19:52:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4579 [2026-10-20 19:52:45][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,579 | MFU: 1.8% [2026-10-20 19:52:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5442 [2026-10-20 19:52:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3578 [2026-10-20 19:52:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9462 [2026-10-20 19:52:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3513 [2026-10-20 19:52:50][INFO][training.pretrain_engine] [Ph2|Step 631] loss=5.6497 | ppl=284.20 | aux=0.02509 | lr=3.00e-04 | gnorm=0.934 | tok/s=29,228 | MFU=7.0% | step=143505ms | tokens=0.669B | mem=4.2/32.7GB | cost=$331.63/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 12:04 (136.2h) | Phase3: SKIPPED | SFT: Oct 26 16:50 (141.0h) | DPO (Final): Oct 26 17:24 (141.5h) [2026-10-20 19:52:50][INFO][training.pretrain_engine] Starting global step 632 (accumulating 32 micro-batches)... [2026-10-20 19:52:50][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 19:53:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7403 [2026-10-20 19:53:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6227 [2026-10-20 19:53:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5665 [2026-10-20 19:53:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8167 [2026-10-20 19:53:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2485 [2026-10-20 19:53:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5261 [2026-10-20 19:53:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4726 [2026-10-20 19:53:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7500 [2026-10-20 19:53:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5188 [2026-10-20 19:53:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4598 [2026-10-20 19:53:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9296 [2026-10-20 19:53:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5420 [2026-10-20 19:54:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5114 [2026-10-20 19:54:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0490 [2026-10-20 19:54:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1118 [2026-10-20 19:54:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1362 [2026-10-20 19:54:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6047 [2026-10-20 19:54:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8140 [2026-10-20 19:54:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6367 [2026-10-20 19:54:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7304 [2026-10-20 19:54:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7841 [2026-10-20 19:54:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7461 [2026-10-20 19:54:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1500 [2026-10-20 19:54:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9745 [2026-10-20 19:54:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5142 [2026-10-20 19:54:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3750 [2026-10-20 19:54:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2899 [2026-10-20 19:54:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2961 [2026-10-20 19:55:09][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,538 | MFU: 1.8% [2026-10-20 19:55:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6700 [2026-10-20 19:55:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9384 [2026-10-20 19:55:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7085 [2026-10-20 19:55:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6724 [2026-10-20 19:55:14][INFO][training.pretrain_engine] [Ph2|Step 632] loss=5.7954 | ppl=328.77 | aux=0.02508 | lr=3.00e-04 | gnorm=0.879 | tok/s=29,229 | MFU=7.0% | step=143500ms | tokens=0.670B | mem=4.2/32.7GB | cost=$332.16/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 12:06 (136.2h) | Phase3: SKIPPED | SFT: Oct 26 16:51 (140.9h) | DPO (Final): Oct 26 17:25 (141.5h) [2026-10-20 19:55:17][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step632 | Tokens: 0.670B | Spent: $332.17 | Remaining: $67.83 [2026-10-20 19:55:17][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step632 [2026-10-20 19:55:17][INFO][training.pretrain_engine] Starting global step 633 (accumulating 32 micro-batches)... [2026-10-20 19:55:17][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 19:55:17][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 19:55:17][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 19:55:17][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 19:55:17][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step632 to Phase-Technologies/forge-500m-pretrain (path: phase2_step632)... [2026-10-20 19:55:17][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 19:55:27][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 19:55:27][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step632 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 19:55:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.9600 [2026-10-20 19:55:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1404 [2026-10-20 19:55:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8985 [2026-10-20 19:55:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.7199 [2026-10-20 19:55:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7191 [2026-10-20 19:55:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8725 [2026-10-20 19:55:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9468 [2026-10-20 19:55:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6774 [2026-10-20 19:56:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1040 [2026-10-20 19:56:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1047 [2026-10-20 19:56:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8465 [2026-10-20 19:56:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0464 [2026-10-20 19:56:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2473 [2026-10-20 19:56:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1057 [2026-10-20 19:56:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4034 [2026-10-20 19:56:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3536 [2026-10-20 19:56:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1159 [2026-10-20 19:56:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9583 [2026-10-20 19:56:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9007 [2026-10-20 19:56:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9933 [2026-10-20 19:57:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7563 [2026-10-20 19:57:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7665 [2026-10-20 19:57:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4125 [2026-10-20 19:57:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1758 [2026-10-20 19:57:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0527 [2026-10-20 19:57:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5466 [2026-10-20 19:57:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0571 [2026-10-20 19:57:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4516 [2026-10-20 19:57:36][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,544 | MFU: 1.8% [2026-10-20 19:57:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7069 [2026-10-20 19:57:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4045 [2026-10-20 19:57:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7652 [2026-10-20 19:57:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5098 [2026-10-20 19:57:41][INFO][training.pretrain_engine] [Ph2|Step 633] loss=5.8194 | ppl=336.78 | aux=0.02509 | lr=3.00e-04 | gnorm=0.750 | tok/s=29,230 | MFU=7.0% | step=143495ms | tokens=0.671B | mem=4.2/32.7GB | cost=$332.70/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 12:07 (136.2h) | Phase3: SKIPPED | SFT: Oct 26 16:53 (140.9h) | DPO (Final): Oct 26 17:27 (141.5h) [2026-10-20 19:57:41][INFO][training.pretrain_engine] Starting global step 634 (accumulating 32 micro-batches)... [2026-10-20 19:57:41][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 19:57:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4308 [2026-10-20 19:57:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4077 [2026-10-20 19:57:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8790 [2026-10-20 19:57:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7099 [2026-10-20 19:58:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9787 [2026-10-20 19:58:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6655 [2026-10-20 19:58:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4869 [2026-10-20 19:58:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0150 [2026-10-20 19:58:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4145 [2026-10-20 19:58:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4210 [2026-10-20 19:58:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9430 [2026-10-20 19:58:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3637 [2026-10-20 19:58:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2904 [2026-10-20 19:58:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1779 [2026-10-20 19:58:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7582 [2026-10-20 19:58:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4638 [2026-10-20 19:59:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0750 [2026-10-20 19:59:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0172 [2026-10-20 19:59:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7169 [2026-10-20 19:59:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1337 [2026-10-20 19:59:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8961 [2026-10-20 19:59:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7027 [2026-10-20 19:59:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6361 [2026-10-20 19:59:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9425 [2026-10-20 19:59:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6501 [2026-10-20 19:59:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6441 [2026-10-20 19:59:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9303 [2026-10-20 19:59:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1127 [2026-10-20 19:59:59][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,550 | MFU: 1.8% [2026-10-20 20:00:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7841 [2026-10-20 20:00:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9923 [2026-10-20 20:00:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4388 [2026-10-20 20:00:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0923 [2026-10-20 20:00:04][INFO][training.pretrain_engine] [Ph2|Step 634] loss=5.7122 | ppl=302.53 | aux=0.02508 | lr=3.00e-04 | gnorm=0.727 | tok/s=29,231 | MFU=7.0% | step=143490ms | tokens=0.672B | mem=4.2/32.7GB | cost=$333.22/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 12:09 (136.2h) | Phase3: SKIPPED | SFT: Oct 26 16:54 (140.9h) | DPO (Final): Oct 26 17:28 (141.5h) [2026-10-20 20:00:04][INFO][training.pretrain_engine] Starting global step 635 (accumulating 32 micro-batches)... [2026-10-20 20:00:04][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 20:00:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8577 [2026-10-20 20:00:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0932 [2026-10-20 20:00:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3922 [2026-10-20 20:00:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0350 [2026-10-20 20:00:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2025 [2026-10-20 20:00:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6032 [2026-10-20 20:00:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8342 [2026-10-20 20:00:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1284 [2026-10-20 20:00:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1516 [2026-10-20 20:00:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9301 [2026-10-20 20:00:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5827 [2026-10-20 20:00:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5576 [2026-10-20 20:01:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9412 [2026-10-20 20:01:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2395 [2026-10-20 20:01:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8736 [2026-10-20 20:01:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9214 [2026-10-20 20:01:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8073 [2026-10-20 20:01:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4775 [2026-10-20 20:01:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1618 [2026-10-20 20:01:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2651 [2026-10-20 20:01:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9956 [2026-10-20 20:01:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8228 [2026-10-20 20:01:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6836 [2026-10-20 20:01:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7805 [2026-10-20 20:02:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6444 [2026-10-20 20:02:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2936 [2026-10-20 20:02:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9599 [2026-10-20 20:02:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3157 [2026-10-20 20:02:23][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,532 | MFU: 1.8% [2026-10-20 20:02:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1258 [2026-10-20 20:02:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6327 [2026-10-20 20:02:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3957 [2026-10-20 20:02:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5597 [2026-10-20 20:02:28][INFO][training.pretrain_engine] [Ph2|Step 635] loss=5.6906 | ppl=296.07 | aux=0.02508 | lr=3.00e-04 | gnorm=0.660 | tok/s=29,231 | MFU=7.0% | step=143486ms | tokens=0.673B | mem=4.2/32.7GB | cost=$333.75/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 12:11 (136.1h) | Phase3: SKIPPED | SFT: Oct 26 16:56 (140.9h) | DPO (Final): Oct 26 17:30 (141.5h) [2026-10-20 20:02:28][INFO][training.pretrain_engine] Starting global step 636 (accumulating 32 micro-batches)... [2026-10-20 20:02:28][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 20:02:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8694 [2026-10-20 20:02:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5751 [2026-10-20 20:02:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8563 [2026-10-20 20:02:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4770 [2026-10-20 20:03:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5662 [2026-10-20 20:03:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5540 [2026-10-20 20:03:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3385 [2026-10-20 20:03:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5872 [2026-10-20 20:03:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2599 [2026-10-20 20:03:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4651 [2026-10-20 20:03:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4446 [2026-10-20 20:03:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3632 [2026-10-20 20:03:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1920 [2026-10-20 20:03:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3273 [2026-10-20 20:03:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9930 [2026-10-20 20:03:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2413 [2026-10-20 20:03:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5874 [2026-10-20 20:03:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1870 [2026-10-20 20:03:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.7491 [2026-10-20 20:03:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5264 [2026-10-20 20:04:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6749 [2026-10-20 20:04:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7028 [2026-10-20 20:04:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7246 [2026-10-20 20:04:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6487 [2026-10-20 20:04:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8326 [2026-10-20 20:04:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8093 [2026-10-20 20:04:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8764 [2026-10-20 20:04:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6407 [2026-10-20 20:04:47][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,567 | MFU: 1.8% [2026-10-20 20:04:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3232 [2026-10-20 20:04:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2321 [2026-10-20 20:04:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5129 [2026-10-20 20:04:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5744 [2026-10-20 20:04:51][INFO][training.pretrain_engine] [Ph2|Step 636] loss=5.6854 | ppl=294.53 | aux=0.02508 | lr=3.00e-04 | gnorm=0.617 | tok/s=29,232 | MFU=7.0% | step=143483ms | tokens=0.674B | mem=4.2/32.7GB | cost=$334.27/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 12:12 (136.1h) | Phase3: SKIPPED | SFT: Oct 26 16:57 (140.9h) | DPO (Final): Oct 26 17:31 (141.5h) [2026-10-20 20:04:51][INFO][training.pretrain_engine] Starting global step 637 (accumulating 32 micro-batches)... [2026-10-20 20:04:51][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 20:05:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5225 [2026-10-20 20:05:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5644 [2026-10-20 20:05:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3352 [2026-10-20 20:05:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8864 [2026-10-20 20:05:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2044 [2026-10-20 20:05:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6335 [2026-10-20 20:05:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0087 [2026-10-20 20:05:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8085 [2026-10-20 20:05:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5688 [2026-10-20 20:05:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6811 [2026-10-20 20:05:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5884 [2026-10-20 20:05:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9671 [2026-10-20 20:06:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8706 [2026-10-20 20:06:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2067 [2026-10-20 20:06:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6459 [2026-10-20 20:06:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0258 [2026-10-20 20:06:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6694 [2026-10-20 20:06:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5518 [2026-10-20 20:06:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7473 [2026-10-20 20:06:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7389 [2026-10-20 20:06:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5353 [2026-10-20 20:06:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3569 [2026-10-20 20:06:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3891 [2026-10-20 20:06:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6199 [2026-10-20 20:06:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3219 [2026-10-20 20:06:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6589 [2026-10-20 20:06:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8279 [2026-10-20 20:06:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6597 [2026-10-20 20:07:10][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,575 | MFU: 1.8% [2026-10-20 20:07:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4849 [2026-10-20 20:07:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8191 [2026-10-20 20:07:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4544 [2026-10-20 20:07:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2688 [2026-10-20 20:07:14][INFO][training.pretrain_engine] [Ph2|Step 637] loss=5.5523 | ppl=257.84 | aux=0.02509 | lr=3.00e-04 | gnorm=0.699 | tok/s=29,233 | MFU=7.0% | step=143476ms | tokens=0.675B | mem=4.2/32.7GB | cost=$334.79/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 12:14 (136.1h) | Phase3: SKIPPED | SFT: Oct 26 16:59 (140.9h) | DPO (Final): Oct 26 17:33 (141.4h) [2026-10-20 20:07:14][INFO][training.pretrain_engine] Starting global step 638 (accumulating 32 micro-batches)... [2026-10-20 20:07:14][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 20:07:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7016 [2026-10-20 20:07:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7402 [2026-10-20 20:07:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8648 [2026-10-20 20:07:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9477 [2026-10-20 20:07:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6796 [2026-10-20 20:07:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4684 [2026-10-20 20:07:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1786 [2026-10-20 20:07:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0347 [2026-10-20 20:08:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5439 [2026-10-20 20:08:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8118 [2026-10-20 20:08:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8938 [2026-10-20 20:08:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0281 [2026-10-20 20:08:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4978 [2026-10-20 20:08:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4068 [2026-10-20 20:08:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5998 [2026-10-20 20:08:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5413 [2026-10-20 20:08:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4025 [2026-10-20 20:08:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9832 [2026-10-20 20:08:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4051 [2026-10-20 20:08:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6499 [2026-10-20 20:09:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2638 [2026-10-20 20:09:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9688 [2026-10-20 20:09:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6901 [2026-10-20 20:09:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5985 [2026-10-20 20:09:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5036 [2026-10-20 20:09:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2138 [2026-10-20 20:09:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0207 [2026-10-20 20:09:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6756 [2026-10-20 20:09:32][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,591 | MFU: 1.8% [2026-10-20 20:09:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8017 [2026-10-20 20:09:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2955 [2026-10-20 20:09:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4971 [2026-10-20 20:09:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0006 [2026-10-20 20:09:37][INFO][training.pretrain_engine] [Ph2|Step 638] loss=5.7066 | ppl=300.85 | aux=0.02508 | lr=3.00e-04 | gnorm=0.695 | tok/s=29,235 | MFU=7.0% | step=143468ms | tokens=0.676B | mem=4.2/32.7GB | cost=$335.31/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 12:15 (136.1h) | Phase3: SKIPPED | SFT: Oct 26 17:00 (140.8h) | DPO (Final): Oct 26 17:34 (141.4h) [2026-10-20 20:09:37][INFO][training.pretrain_engine] Starting global step 639 (accumulating 32 micro-batches)... [2026-10-20 20:09:37][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 20:09:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3955 [2026-10-20 20:09:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7221 [2026-10-20 20:09:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6322 [2026-10-20 20:09:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2983 [2026-10-20 20:10:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7174 [2026-10-20 20:10:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6255 [2026-10-20 20:10:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6434 [2026-10-20 20:10:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6784 [2026-10-20 20:10:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3413 [2026-10-20 20:10:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5386 [2026-10-20 20:10:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9099 [2026-10-20 20:10:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6861 [2026-10-20 20:10:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6401 [2026-10-20 20:10:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6618 [2026-10-20 20:10:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5998 [2026-10-20 20:10:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7658 [2026-10-20 20:11:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0798 [2026-10-20 20:11:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3107 [2026-10-20 20:11:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4134 [2026-10-20 20:11:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2278 [2026-10-20 20:11:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5160 [2026-10-20 20:11:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4177 [2026-10-20 20:11:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5923 [2026-10-20 20:11:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5767 [2026-10-20 20:11:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0888 [2026-10-20 20:11:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0848 [2026-10-20 20:11:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3728 [2026-10-20 20:11:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7184 [2026-10-20 20:11:55][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,606 | MFU: 1.8% [2026-10-20 20:12:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9785 [2026-10-20 20:12:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6782 [2026-10-20 20:12:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5953 [2026-10-20 20:12:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6210 [2026-10-20 20:12:00][INFO][training.pretrain_engine] [Ph2|Step 639] loss=5.4605 | ppl=235.23 | aux=0.02508 | lr=3.00e-04 | gnorm=0.574 | tok/s=29,235 | MFU=7.0% | step=143469ms | tokens=0.677B | mem=4.2/32.7GB | cost=$335.84/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 12:17 (136.1h) | Phase3: SKIPPED | SFT: Oct 26 17:02 (140.8h) | DPO (Final): Oct 26 17:36 (141.4h) [2026-10-20 20:12:00][INFO][training.pretrain_engine] Starting global step 640 (accumulating 32 micro-batches)... [2026-10-20 20:12:00][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 20:12:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3579 [2026-10-20 20:12:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8906 [2026-10-20 20:12:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2780 [2026-10-20 20:12:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2019 [2026-10-20 20:12:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6020 [2026-10-20 20:12:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5645 [2026-10-20 20:12:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2867 [2026-10-20 20:12:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3193 [2026-10-20 20:12:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7055 [2026-10-20 20:12:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2775 [2026-10-20 20:12:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4861 [2026-10-20 20:12:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6209 [2026-10-20 20:13:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5725 [2026-10-20 20:13:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5051 [2026-10-20 20:13:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1642 [2026-10-20 20:13:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6679 [2026-10-20 20:13:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0908 [2026-10-20 20:13:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1225 [2026-10-20 20:13:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2111 [2026-10-20 20:13:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8326 [2026-10-20 20:13:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1906 [2026-10-20 20:13:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2164 [2026-10-20 20:13:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1462 [2026-10-20 20:13:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5304 [2026-10-20 20:14:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1856 [2026-10-20 20:14:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8594 [2026-10-20 20:14:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1445 [2026-10-20 20:14:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7760 [2026-10-20 20:14:18][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,594 | MFU: 1.8% [2026-10-20 20:14:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4413 [2026-10-20 20:14:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1504 [2026-10-20 20:14:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6263 [2026-10-20 20:14:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7241 [2026-10-20 20:14:23][INFO][training.pretrain_engine] [Ph2|Step 640] loss=5.6133 | ppl=274.05 | aux=0.02508 | lr=3.00e-04 | gnorm=0.547 | tok/s=29,235 | MFU=7.0% | step=143468ms | tokens=0.678B | mem=4.2/32.7GB | cost=$336.36/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 12:19 (136.1h) | Phase3: SKIPPED | SFT: Oct 26 17:04 (140.8h) | DPO (Final): Oct 26 17:38 (141.4h) [2026-10-20 20:14:23][INFO][training.pretrain_engine] Starting global step 641 (accumulating 32 micro-batches)... [2026-10-20 20:14:23][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 20:14:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7233 [2026-10-20 20:14:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6007 [2026-10-20 20:14:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2671 [2026-10-20 20:14:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2147 [2026-10-20 20:14:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2338 [2026-10-20 20:14:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8722 [2026-10-20 20:14:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6252 [2026-10-20 20:14:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8939 [2026-10-20 20:15:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4265 [2026-10-20 20:15:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3382 [2026-10-20 20:15:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6078 [2026-10-20 20:15:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5760 [2026-10-20 20:15:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1047 [2026-10-20 20:15:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0996 [2026-10-20 20:15:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8856 [2026-10-20 20:15:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0548 [2026-10-20 20:15:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5904 [2026-10-20 20:15:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1383 [2026-10-20 20:15:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7054 [2026-10-20 20:15:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5736 [2026-10-20 20:16:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8293 [2026-10-20 20:16:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5160 [2026-10-20 20:16:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7901 [2026-10-20 20:16:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7752 [2026-10-20 20:16:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0829 [2026-10-20 20:16:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4931 [2026-10-20 20:16:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7206 [2026-10-20 20:16:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6066 [2026-10-20 20:16:42][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,572 | MFU: 1.8% [2026-10-20 20:16:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3079 [2026-10-20 20:16:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2753 [2026-10-20 20:16:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4618 [2026-10-20 20:16:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5675 [2026-10-20 20:16:46][INFO][training.pretrain_engine] [Ph2|Step 641] loss=5.6165 | ppl=274.92 | aux=0.02509 | lr=3.00e-04 | gnorm=0.574 | tok/s=29,237 | MFU=7.0% | step=143461ms | tokens=0.679B | mem=4.2/32.7GB | cost=$336.88/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 12:20 (136.1h) | Phase3: SKIPPED | SFT: Oct 26 17:05 (140.8h) | DPO (Final): Oct 26 17:39 (141.4h) [2026-10-20 20:16:50][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step641 | Tokens: 0.679B | Spent: $336.90 | Remaining: $63.10 [2026-10-20 20:16:50][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step641 [2026-10-20 20:16:50][INFO][training.pretrain_engine] Starting global step 642 (accumulating 32 micro-batches)... [2026-10-20 20:16:50][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 20:16:50][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 20:16:50][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 20:16:50][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 20:16:50][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step641 to Phase-Technologies/forge-500m-pretrain (path: phase2_step641)... [2026-10-20 20:16:50][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 20:17:00][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 20:17:00][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step641 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 20:17:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2233 [2026-10-20 20:17:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4861 [2026-10-20 20:17:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7307 [2026-10-20 20:17:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2224 [2026-10-20 20:17:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5533 [2026-10-20 20:17:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5875 [2026-10-20 20:17:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5510 [2026-10-20 20:17:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1325 [2026-10-20 20:17:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6107 [2026-10-20 20:17:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7742 [2026-10-20 20:17:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5301 [2026-10-20 20:17:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6218 [2026-10-20 20:17:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0253 [2026-10-20 20:17:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8250 [2026-10-20 20:17:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0353 [2026-10-20 20:18:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6820 [2026-10-20 20:18:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8642 [2026-10-20 20:18:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5274 [2026-10-20 20:18:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6737 [2026-10-20 20:18:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6690 [2026-10-20 20:18:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5243 [2026-10-20 20:18:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3800 [2026-10-20 20:18:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6607 [2026-10-20 20:18:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5729 [2026-10-20 20:18:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5924 [2026-10-20 20:18:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2518 [2026-10-20 20:18:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4548 [2026-10-20 20:18:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7810 [2026-10-20 20:19:09][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,541 | MFU: 1.8% [2026-10-20 20:19:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5617 [2026-10-20 20:19:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7261 [2026-10-20 20:19:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8879 [2026-10-20 20:19:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7734 [2026-10-20 20:19:13][INFO][training.pretrain_engine] [Ph2|Step 642] loss=5.6597 | ppl=287.08 | aux=0.02509 | lr=3.00e-04 | gnorm=0.547 | tok/s=29,238 | MFU=7.0% | step=143456ms | tokens=0.681B | mem=4.2/32.7GB | cost=$337.42/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 12:21 (136.0h) | Phase3: SKIPPED | SFT: Oct 26 17:06 (140.8h) | DPO (Final): Oct 26 17:41 (141.4h) [2026-10-20 20:19:13][INFO][training.pretrain_engine] Starting global step 643 (accumulating 32 micro-batches)... [2026-10-20 20:19:13][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 20:19:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9495 [2026-10-20 20:19:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8628 [2026-10-20 20:19:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9455 [2026-10-20 20:19:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2687 [2026-10-20 20:19:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4366 [2026-10-20 20:19:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7374 [2026-10-20 20:19:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0710 [2026-10-20 20:19:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4413 [2026-10-20 20:20:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0215 [2026-10-20 20:20:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1618 [2026-10-20 20:20:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7958 [2026-10-20 20:20:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0414 [2026-10-20 20:20:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9248 [2026-10-20 20:20:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5493 [2026-10-20 20:20:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7372 [2026-10-20 20:20:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9627 [2026-10-20 20:20:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5685 [2026-10-20 20:20:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5059 [2026-10-20 20:20:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5351 [2026-10-20 20:20:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5060 [2026-10-20 20:21:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5133 [2026-10-20 20:21:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2574 [2026-10-20 20:21:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4154 [2026-10-20 20:21:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3656 [2026-10-20 20:21:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3550 [2026-10-20 20:21:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3031 [2026-10-20 20:21:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0323 [2026-10-20 20:21:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1764 [2026-10-20 20:21:32][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,549 | MFU: 1.8% [2026-10-20 20:21:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6196 [2026-10-20 20:21:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6695 [2026-10-20 20:21:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5951 [2026-10-20 20:21:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2356 [2026-10-20 20:21:37][INFO][training.pretrain_engine] [Ph2|Step 643] loss=5.5230 | ppl=250.39 | aux=0.02509 | lr=3.00e-04 | gnorm=0.590 | tok/s=29,238 | MFU=7.0% | step=143453ms | tokens=0.682B | mem=4.2/32.7GB | cost=$337.94/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 12:23 (136.0h) | Phase3: SKIPPED | SFT: Oct 26 17:08 (140.8h) | DPO (Final): Oct 26 17:42 (141.4h) [2026-10-20 20:21:37][INFO][training.pretrain_engine] Starting global step 644 (accumulating 32 micro-batches)... [2026-10-20 20:21:37][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 20:21:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6986 [2026-10-20 20:21:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6893 [2026-10-20 20:21:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9580 [2026-10-20 20:21:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7989 [2026-10-20 20:22:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8151 [2026-10-20 20:22:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7134 [2026-10-20 20:22:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6686 [2026-10-20 20:22:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4364 [2026-10-20 20:22:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7959 [2026-10-20 20:22:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9101 [2026-10-20 20:22:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6342 [2026-10-20 20:22:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8477 [2026-10-20 20:22:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3605 [2026-10-20 20:22:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8305 [2026-10-20 20:22:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3903 [2026-10-20 20:22:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7295 [2026-10-20 20:23:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1559 [2026-10-20 20:23:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2040 [2026-10-20 20:23:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6362 [2026-10-20 20:23:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8801 [2026-10-20 20:23:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5186 [2026-10-20 20:23:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7524 [2026-10-20 20:23:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8306 [2026-10-20 20:23:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7192 [2026-10-20 20:23:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7686 [2026-10-20 20:23:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9738 [2026-10-20 20:23:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4238 [2026-10-20 20:23:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6340 [2026-10-20 20:23:55][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,553 | MFU: 1.8% [2026-10-20 20:24:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2364 [2026-10-20 20:24:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4599 [2026-10-20 20:24:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3934 [2026-10-20 20:24:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5824 [2026-10-20 20:24:00][INFO][training.pretrain_engine] [Ph2|Step 644] loss=5.6126 | ppl=273.86 | aux=0.02507 | lr=3.00e-04 | gnorm=0.543 | tok/s=29,239 | MFU=7.0% | step=143447ms | tokens=0.683B | mem=4.2/32.7GB | cost=$338.47/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 12:25 (136.0h) | Phase3: SKIPPED | SFT: Oct 26 17:10 (140.8h) | DPO (Final): Oct 26 17:44 (141.3h) [2026-10-20 20:24:00][INFO][training.pretrain_engine] Starting global step 645 (accumulating 32 micro-batches)... [2026-10-20 20:24:00][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 20:24:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7433 [2026-10-20 20:24:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5054 [2026-10-20 20:24:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5624 [2026-10-20 20:24:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7973 [2026-10-20 20:24:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4045 [2026-10-20 20:24:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0460 [2026-10-20 20:24:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4651 [2026-10-20 20:24:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2821 [2026-10-20 20:24:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3276 [2026-10-20 20:24:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5066 [2026-10-20 20:24:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1760 [2026-10-20 20:24:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7980 [2026-10-20 20:25:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6708 [2026-10-20 20:25:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7871 [2026-10-20 20:25:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5300 [2026-10-20 20:25:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8723 [2026-10-20 20:25:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7372 [2026-10-20 20:25:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5140 [2026-10-20 20:25:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3441 [2026-10-20 20:25:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0269 [2026-10-20 20:25:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8771 [2026-10-20 20:25:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8244 [2026-10-20 20:25:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6826 [2026-10-20 20:25:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0309 [2026-10-20 20:26:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8391 [2026-10-20 20:26:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6332 [2026-10-20 20:26:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0971 [2026-10-20 20:26:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3391 [2026-10-20 20:26:18][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,576 | MFU: 1.8% [2026-10-20 20:26:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8365 [2026-10-20 20:26:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7902 [2026-10-20 20:26:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4792 [2026-10-20 20:26:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6119 [2026-10-20 20:26:23][INFO][training.pretrain_engine] [Ph2|Step 645] loss=5.6262 | ppl=277.60 | aux=0.02508 | lr=3.00e-04 | gnorm=0.562 | tok/s=29,241 | MFU=7.0% | step=143440ms | tokens=0.684B | mem=4.2/32.7GB | cost=$338.99/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 12:26 (136.0h) | Phase3: SKIPPED | SFT: Oct 26 17:11 (140.7h) | DPO (Final): Oct 26 17:45 (141.3h) [2026-10-20 20:26:23][INFO][training.pretrain_engine] Starting global step 646 (accumulating 32 micro-batches)... [2026-10-20 20:26:23][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 20:26:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7310 [2026-10-20 20:26:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2793 [2026-10-20 20:26:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5562 [2026-10-20 20:26:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6455 [2026-10-20 20:26:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7424 [2026-10-20 20:26:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6985 [2026-10-20 20:26:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7588 [2026-10-20 20:26:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7474 [2026-10-20 20:27:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5013 [2026-10-20 20:27:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4738 [2026-10-20 20:27:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0404 [2026-10-20 20:27:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8150 [2026-10-20 20:27:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3408 [2026-10-20 20:27:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0358 [2026-10-20 20:27:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3311 [2026-10-20 20:27:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9179 [2026-10-20 20:27:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4854 [2026-10-20 20:27:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4519 [2026-10-20 20:27:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4224 [2026-10-20 20:27:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5781 [2026-10-20 20:28:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5432 [2026-10-20 20:28:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5241 [2026-10-20 20:28:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1127 [2026-10-20 20:28:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2006 [2026-10-20 20:28:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8766 [2026-10-20 20:28:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4263 [2026-10-20 20:28:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3838 [2026-10-20 20:28:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6479 [2026-10-20 20:28:42][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,570 | MFU: 1.8% [2026-10-20 20:28:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7275 [2026-10-20 20:28:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7828 [2026-10-20 20:28:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7755 [2026-10-20 20:28:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8487 [2026-10-20 20:28:46][INFO][training.pretrain_engine] [Ph2|Step 646] loss=5.5582 | ppl=259.35 | aux=0.02507 | lr=3.00e-04 | gnorm=0.586 | tok/s=29,241 | MFU=7.0% | step=143438ms | tokens=0.685B | mem=4.2/32.7GB | cost=$339.51/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 12:28 (136.0h) | Phase3: SKIPPED | SFT: Oct 26 17:13 (140.7h) | DPO (Final): Oct 26 17:47 (141.3h) [2026-10-20 20:28:46][INFO][training.pretrain_engine] Starting global step 647 (accumulating 32 micro-batches)... [2026-10-20 20:28:46][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 20:29:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4730 [2026-10-20 20:29:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6597 [2026-10-20 20:29:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1224 [2026-10-20 20:29:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9716 [2026-10-20 20:29:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6786 [2026-10-20 20:29:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3778 [2026-10-20 20:29:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8525 [2026-10-20 20:29:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4895 [2026-10-20 20:29:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6678 [2026-10-20 20:29:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7968 [2026-10-20 20:29:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4907 [2026-10-20 20:29:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6618 [2026-10-20 20:29:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9896 [2026-10-20 20:29:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1406 [2026-10-20 20:29:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1889 [2026-10-20 20:29:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9122 [2026-10-20 20:30:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4279 [2026-10-20 20:30:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7786 [2026-10-20 20:30:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7942 [2026-10-20 20:30:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4257 [2026-10-20 20:30:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7398 [2026-10-20 20:30:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3221 [2026-10-20 20:30:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1036 [2026-10-20 20:30:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5533 [2026-10-20 20:30:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7897 [2026-10-20 20:30:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0986 [2026-10-20 20:30:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5999 [2026-10-20 20:30:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0472 [2026-10-20 20:31:04][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,591 | MFU: 1.8% [2026-10-20 20:31:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0795 [2026-10-20 20:31:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8885 [2026-10-20 20:31:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8709 [2026-10-20 20:31:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0041 [2026-10-20 20:31:09][INFO][training.pretrain_engine] [Ph2|Step 647] loss=5.5062 | ppl=246.22 | aux=0.02509 | lr=3.00e-04 | gnorm=0.703 | tok/s=29,241 | MFU=7.0% | step=143439ms | tokens=0.686B | mem=4.2/32.7GB | cost=$340.04/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 12:29 (136.0h) | Phase3: SKIPPED | SFT: Oct 26 17:14 (140.7h) | DPO (Final): Oct 26 17:49 (141.3h) [2026-10-20 20:31:09][INFO][training.pretrain_engine] Starting global step 648 (accumulating 32 micro-batches)... [2026-10-20 20:31:09][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 20:31:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5160 [2026-10-20 20:31:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8812 [2026-10-20 20:31:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0247 [2026-10-20 20:31:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8946 [2026-10-20 20:31:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9511 [2026-10-20 20:31:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0129 [2026-10-20 20:31:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4033 [2026-10-20 20:31:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7896 [2026-10-20 20:32:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9527 [2026-10-20 20:32:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8625 [2026-10-20 20:32:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5630 [2026-10-20 20:32:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4718 [2026-10-20 20:32:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5613 [2026-10-20 20:32:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5200 [2026-10-20 20:32:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9818 [2026-10-20 20:32:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3931 [2026-10-20 20:32:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7707 [2026-10-20 20:32:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5427 [2026-10-20 20:32:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0694 [2026-10-20 20:32:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7251 [2026-10-20 20:32:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4555 [2026-10-20 20:32:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0701 [2026-10-20 20:32:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6891 [2026-10-20 20:32:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2881 [2026-10-20 20:33:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4867 [2026-10-20 20:33:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6562 [2026-10-20 20:33:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7514 [2026-10-20 20:33:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7920 [2026-10-20 20:33:27][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,596 | MFU: 1.8% [2026-10-20 20:33:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3989 [2026-10-20 20:33:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1874 [2026-10-20 20:33:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4292 [2026-10-20 20:33:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4119 [2026-10-20 20:33:32][INFO][training.pretrain_engine] [Ph2|Step 648] loss=5.6730 | ppl=290.90 | aux=0.02508 | lr=3.00e-04 | gnorm=0.766 | tok/s=29,241 | MFU=7.0% | step=143437ms | tokens=0.687B | mem=4.2/32.7GB | cost=$340.56/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 12:31 (136.0h) | Phase3: SKIPPED | SFT: Oct 26 17:16 (140.7h) | DPO (Final): Oct 26 17:50 (141.3h) [2026-10-20 20:33:32][INFO][training.pretrain_engine] Starting global step 649 (accumulating 32 micro-batches)... [2026-10-20 20:33:32][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 20:33:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4092 [2026-10-20 20:33:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4294 [2026-10-20 20:33:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6466 [2026-10-20 20:33:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4740 [2026-10-20 20:34:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5784 [2026-10-20 20:34:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7019 [2026-10-20 20:34:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8273 [2026-10-20 20:34:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6266 [2026-10-20 20:34:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9547 [2026-10-20 20:34:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1892 [2026-10-20 20:34:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5653 [2026-10-20 20:34:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5588 [2026-10-20 20:34:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7641 [2026-10-20 20:34:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7268 [2026-10-20 20:34:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7747 [2026-10-20 20:34:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5102 [2026-10-20 20:35:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0364 [2026-10-20 20:35:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4920 [2026-10-20 20:35:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0786 [2026-10-20 20:35:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0685 [2026-10-20 20:35:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5686 [2026-10-20 20:35:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5861 [2026-10-20 20:35:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8316 [2026-10-20 20:35:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7886 [2026-10-20 20:35:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5389 [2026-10-20 20:35:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5230 [2026-10-20 20:35:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4140 [2026-10-20 20:35:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7651 [2026-10-20 20:35:50][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,586 | MFU: 1.8% [2026-10-20 20:35:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0073 [2026-10-20 20:35:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6660 [2026-10-20 20:35:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0571 [2026-10-20 20:35:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5253 [2026-10-20 20:35:55][INFO][training.pretrain_engine] [Ph2|Step 649] loss=5.7480 | ppl=313.55 | aux=0.02508 | lr=3.00e-04 | gnorm=0.969 | tok/s=29,242 | MFU=7.0% | step=143436ms | tokens=0.688B | mem=4.2/32.7GB | cost=$341.08/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 12:33 (136.0h) | Phase3: SKIPPED | SFT: Oct 26 17:18 (140.7h) | DPO (Final): Oct 26 17:52 (141.3h) [2026-10-20 20:35:55][INFO][training.pretrain_engine] Starting global step 650 (accumulating 32 micro-batches)... [2026-10-20 20:35:55][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 20:36:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5136 [2026-10-20 20:36:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4562 [2026-10-20 20:36:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3729 [2026-10-20 20:36:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0709 [2026-10-20 20:36:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6766 [2026-10-20 20:36:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0399 [2026-10-20 20:36:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0074 [2026-10-20 20:36:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3969 [2026-10-20 20:36:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4739 [2026-10-20 20:36:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5150 [2026-10-20 20:36:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8497 [2026-10-20 20:36:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5299 [2026-10-20 20:37:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8494 [2026-10-20 20:37:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2349 [2026-10-20 20:37:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8207 [2026-10-20 20:37:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5311 [2026-10-20 20:37:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9012 [2026-10-20 20:37:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0492 [2026-10-20 20:37:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6037 [2026-10-20 20:37:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1490 [2026-10-20 20:37:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3480 [2026-10-20 20:37:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0307 [2026-10-20 20:37:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4858 [2026-10-20 20:37:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6162 [2026-10-20 20:38:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8131 [2026-10-20 20:38:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6760 [2026-10-20 20:38:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2136 [2026-10-20 20:38:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1591 [2026-10-20 20:38:13][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,574 | MFU: 1.8% [2026-10-20 20:38:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7147 [2026-10-20 20:38:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4740 [2026-10-20 20:38:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5149 [2026-10-20 20:38:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2902 [2026-10-20 20:38:18][INFO][training.pretrain_engine] [Ph2|Step 650] loss=5.6356 | ppl=280.23 | aux=0.02507 | lr=3.00e-04 | gnorm=1.516 | tok/s=29,241 | MFU=7.0% | step=143438ms | tokens=0.689B | mem=4.2/32.7GB | cost=$341.61/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 12:35 (135.9h) | Phase3: SKIPPED | SFT: Oct 26 17:20 (140.7h) | DPO (Final): Oct 26 17:54 (141.3h) [2026-10-20 20:38:21][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step650 | Tokens: 0.689B | Spent: $341.62 | Remaining: $58.38 [2026-10-20 20:38:21][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step650 [2026-10-20 20:38:21][INFO][training.pretrain_engine] Starting global step 651 (accumulating 32 micro-batches)... [2026-10-20 20:38:21][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 20:38:21][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 20:38:21][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 20:38:21][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 20:38:21][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step650 to Phase-Technologies/forge-500m-pretrain (path: phase2_step650)... [2026-10-20 20:38:22][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 20:38:31][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 20:38:31][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step650 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 20:38:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2309 [2026-10-20 20:38:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2531 [2026-10-20 20:38:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1841 [2026-10-20 20:38:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1446 [2026-10-20 20:38:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0080 [2026-10-20 20:38:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4362 [2026-10-20 20:38:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4304 [2026-10-20 20:38:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9573 [2026-10-20 20:39:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1914 [2026-10-20 20:39:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0935 [2026-10-20 20:39:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4872 [2026-10-20 20:39:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5203 [2026-10-20 20:39:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1102 [2026-10-20 20:39:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4372 [2026-10-20 20:39:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5917 [2026-10-20 20:39:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7371 [2026-10-20 20:39:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5526 [2026-10-20 20:39:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6270 [2026-10-20 20:39:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5354 [2026-10-20 20:39:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4830 [2026-10-20 20:40:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5699 [2026-10-20 20:40:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4243 [2026-10-20 20:40:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2170 [2026-10-20 20:40:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5042 [2026-10-20 20:40:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8810 [2026-10-20 20:40:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3698 [2026-10-20 20:40:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2456 [2026-10-20 20:40:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5373 [2026-10-20 20:40:40][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,543 | MFU: 1.8% [2026-10-20 20:40:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1645 [2026-10-20 20:40:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4763 [2026-10-20 20:40:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0772 [2026-10-20 20:40:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3707 [2026-10-20 20:40:45][INFO][training.pretrain_engine] [Ph2|Step 651] loss=5.5413 | ppl=255.01 | aux=0.02507 | lr=3.00e-04 | gnorm=0.852 | tok/s=29,240 | MFU=7.0% | step=143443ms | tokens=0.690B | mem=4.2/32.7GB | cost=$342.14/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 12:37 (135.9h) | Phase3: SKIPPED | SFT: Oct 26 17:22 (140.7h) | DPO (Final): Oct 26 17:56 (141.3h) [2026-10-20 20:40:45][INFO][training.pretrain_engine] Starting global step 652 (accumulating 32 micro-batches)... [2026-10-20 20:40:45][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 20:41:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3965 [2026-10-20 20:41:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6352 [2026-10-20 20:41:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4672 [2026-10-20 20:41:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1262 [2026-10-20 20:41:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2381 [2026-10-20 20:41:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4171 [2026-10-20 20:41:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7113 [2026-10-20 20:41:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5412 [2026-10-20 20:41:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7919 [2026-10-20 20:41:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4173 [2026-10-20 20:41:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8267 [2026-10-20 20:41:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4375 [2026-10-20 20:41:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7472 [2026-10-20 20:41:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9101 [2026-10-20 20:41:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2901 [2026-10-20 20:41:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0567 [2026-10-20 20:42:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6211 [2026-10-20 20:42:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7543 [2026-10-20 20:42:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5331 [2026-10-20 20:42:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9371 [2026-10-20 20:42:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4072 [2026-10-20 20:42:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4326 [2026-10-20 20:42:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1869 [2026-10-20 20:42:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3265 [2026-10-20 20:42:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6996 [2026-10-20 20:42:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6339 [2026-10-20 20:42:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7662 [2026-10-20 20:42:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0796 [2026-10-20 20:43:04][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,539 | MFU: 1.8% [2026-10-20 20:43:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7608 [2026-10-20 20:43:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6968 [2026-10-20 20:43:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5448 [2026-10-20 20:43:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8110 [2026-10-20 20:43:08][INFO][training.pretrain_engine] [Ph2|Step 652] loss=5.6250 | ppl=277.26 | aux=0.02509 | lr=3.00e-04 | gnorm=1.250 | tok/s=29,240 | MFU=7.0% | step=143444ms | tokens=0.691B | mem=4.2/32.7GB | cost=$342.67/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 12:39 (135.9h) | Phase3: SKIPPED | SFT: Oct 26 17:24 (140.7h) | DPO (Final): Oct 26 17:58 (141.3h) [2026-10-20 20:43:08][INFO][training.pretrain_engine] Starting global step 653 (accumulating 32 micro-batches)... [2026-10-20 20:43:08][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 20:43:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2090 [2026-10-20 20:43:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3815 [2026-10-20 20:43:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6317 [2026-10-20 20:43:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2163 [2026-10-20 20:43:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5849 [2026-10-20 20:43:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1454 [2026-10-20 20:43:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2015 [2026-10-20 20:43:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7781 [2026-10-20 20:44:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2146 [2026-10-20 20:44:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3780 [2026-10-20 20:44:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6183 [2026-10-20 20:44:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5244 [2026-10-20 20:44:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5001 [2026-10-20 20:44:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3612 [2026-10-20 20:44:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5901 [2026-10-20 20:44:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8575 [2026-10-20 20:44:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3475 [2026-10-20 20:44:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2309 [2026-10-20 20:44:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3126 [2026-10-20 20:44:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1295 [2026-10-20 20:44:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8206 [2026-10-20 20:44:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5792 [2026-10-20 20:44:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2871 [2026-10-20 20:44:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4777 [2026-10-20 20:45:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7973 [2026-10-20 20:45:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3766 [2026-10-20 20:45:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3990 [2026-10-20 20:45:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8233 [2026-10-20 20:45:27][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,561 | MFU: 1.8% [2026-10-20 20:45:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4683 [2026-10-20 20:45:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6479 [2026-10-20 20:45:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5785 [2026-10-20 20:45:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8406 [2026-10-20 20:45:32][INFO][training.pretrain_engine] [Ph2|Step 653] loss=5.5517 | ppl=257.68 | aux=0.02507 | lr=3.00e-04 | gnorm=1.039 | tok/s=29,240 | MFU=7.0% | step=143445ms | tokens=0.692B | mem=4.2/32.7GB | cost=$343.19/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 12:41 (135.9h) | Phase3: SKIPPED | SFT: Oct 26 17:26 (140.7h) | DPO (Final): Oct 26 18:00 (141.2h) [2026-10-20 20:45:32][INFO][training.pretrain_engine] Starting global step 654 (accumulating 32 micro-batches)... [2026-10-20 20:45:32][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 20:45:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4631 [2026-10-20 20:45:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2455 [2026-10-20 20:45:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7434 [2026-10-20 20:45:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8730 [2026-10-20 20:46:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5937 [2026-10-20 20:46:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5916 [2026-10-20 20:46:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5460 [2026-10-20 20:46:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4037 [2026-10-20 20:46:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5146 [2026-10-20 20:46:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5176 [2026-10-20 20:46:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1934 [2026-10-20 20:46:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3928 [2026-10-20 20:46:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6634 [2026-10-20 20:46:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0704 [2026-10-20 20:46:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8866 [2026-10-20 20:46:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9215 [2026-10-20 20:47:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6487 [2026-10-20 20:47:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6796 [2026-10-20 20:47:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6027 [2026-10-20 20:47:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3461 [2026-10-20 20:47:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4356 [2026-10-20 20:47:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8251 [2026-10-20 20:47:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5890 [2026-10-20 20:47:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7034 [2026-10-20 20:47:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7560 [2026-10-20 20:47:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5322 [2026-10-20 20:47:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7782 [2026-10-20 20:47:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5052 [2026-10-20 20:47:51][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,545 | MFU: 1.8% [2026-10-20 20:47:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6242 [2026-10-20 20:47:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3596 [2026-10-20 20:47:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8007 [2026-10-20 20:47:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2463 [2026-10-20 20:47:55][INFO][training.pretrain_engine] [Ph2|Step 654] loss=5.5224 | ppl=250.24 | aux=0.02509 | lr=3.00e-04 | gnorm=1.320 | tok/s=29,238 | MFU=7.0% | step=143452ms | tokens=0.693B | mem=4.2/32.7GB | cost=$343.72/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 12:43 (135.9h) | Phase3: SKIPPED | SFT: Oct 26 17:28 (140.7h) | DPO (Final): Oct 26 18:02 (141.2h) [2026-10-20 20:47:55][INFO][training.pretrain_engine] Starting global step 655 (accumulating 32 micro-batches)... [2026-10-20 20:47:55][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 20:48:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5215 [2026-10-20 20:48:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7543 [2026-10-20 20:48:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9920 [2026-10-20 20:48:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5121 [2026-10-20 20:48:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6209 [2026-10-20 20:48:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6997 [2026-10-20 20:48:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0934 [2026-10-20 20:48:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8249 [2026-10-20 20:48:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2873 [2026-10-20 20:48:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4901 [2026-10-20 20:48:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1374 [2026-10-20 20:48:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7654 [2026-10-20 20:49:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4585 [2026-10-20 20:49:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1550 [2026-10-20 20:49:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8047 [2026-10-20 20:49:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8067 [2026-10-20 20:49:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8727 [2026-10-20 20:49:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2140 [2026-10-20 20:49:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3693 [2026-10-20 20:49:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6672 [2026-10-20 20:49:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0501 [2026-10-20 20:49:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.4745 [2026-10-20 20:49:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5914 [2026-10-20 20:49:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6554 [2026-10-20 20:50:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3920 [2026-10-20 20:50:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3646 [2026-10-20 20:50:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7069 [2026-10-20 20:50:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5762 [2026-10-20 20:50:14][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,560 | MFU: 1.8% [2026-10-20 20:50:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7350 [2026-10-20 20:50:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4295 [2026-10-20 20:50:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5981 [2026-10-20 20:50:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9496 [2026-10-20 20:50:18][INFO][training.pretrain_engine] [Ph2|Step 655] loss=5.5643 | ppl=260.95 | aux=0.02508 | lr=3.00e-04 | gnorm=0.684 | tok/s=29,238 | MFU=7.0% | step=143456ms | tokens=0.694B | mem=4.2/32.7GB | cost=$344.24/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 12:45 (135.9h) | Phase3: SKIPPED | SFT: Oct 26 17:30 (140.7h) | DPO (Final): Oct 26 18:04 (141.2h) [2026-10-20 20:50:18][INFO][training.pretrain_engine] Starting global step 656 (accumulating 32 micro-batches)... [2026-10-20 20:50:18][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 20:50:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9595 [2026-10-20 20:50:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9014 [2026-10-20 20:50:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7915 [2026-10-20 20:50:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8759 [2026-10-20 20:50:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1026 [2026-10-20 20:50:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7752 [2026-10-20 20:50:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4706 [2026-10-20 20:50:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5799 [2026-10-20 20:51:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2490 [2026-10-20 20:51:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2393 [2026-10-20 20:51:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1783 [2026-10-20 20:51:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9326 [2026-10-20 20:51:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3533 [2026-10-20 20:51:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8495 [2026-10-20 20:51:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2854 [2026-10-20 20:51:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5142 [2026-10-20 20:51:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7296 [2026-10-20 20:51:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7829 [2026-10-20 20:51:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8968 [2026-10-20 20:51:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8059 [2026-10-20 20:52:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9226 [2026-10-20 20:52:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0859 [2026-10-20 20:52:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3760 [2026-10-20 20:52:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9079 [2026-10-20 20:52:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4424 [2026-10-20 20:52:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9254 [2026-10-20 20:52:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.6642 [2026-10-20 20:52:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9360 [2026-10-20 20:52:37][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,543 | MFU: 1.8% [2026-10-20 20:52:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4678 [2026-10-20 20:52:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5651 [2026-10-20 20:52:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8045 [2026-10-20 20:52:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6220 [2026-10-20 20:52:42][INFO][training.pretrain_engine] [Ph2|Step 656] loss=5.6006 | ppl=270.59 | aux=0.02507 | lr=3.00e-04 | gnorm=0.977 | tok/s=29,238 | MFU=7.0% | step=143452ms | tokens=0.695B | mem=4.2/32.7GB | cost=$344.76/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 12:46 (135.9h) | Phase3: SKIPPED | SFT: Oct 26 17:31 (140.7h) | DPO (Final): Oct 26 18:06 (141.2h) [2026-10-20 20:52:42][INFO][training.pretrain_engine] Starting global step 657 (accumulating 32 micro-batches)... [2026-10-20 20:52:42][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 20:53:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8659 [2026-10-20 20:53:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6468 [2026-10-20 20:53:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2498 [2026-10-20 20:53:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5714 [2026-10-20 20:53:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2728 [2026-10-20 20:53:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5297 [2026-10-20 20:53:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4321 [2026-10-20 20:53:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8723 [2026-10-20 20:53:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3523 [2026-10-20 20:53:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5437 [2026-10-20 20:53:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4030 [2026-10-20 20:53:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2538 [2026-10-20 20:53:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5181 [2026-10-20 20:53:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2272 [2026-10-20 20:53:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2990 [2026-10-20 20:53:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4215 [2026-10-20 20:54:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7533 [2026-10-20 20:54:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4387 [2026-10-20 20:54:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3460 [2026-10-20 20:54:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0518 [2026-10-20 20:54:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6819 [2026-10-20 20:54:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7999 [2026-10-20 20:54:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5063 [2026-10-20 20:54:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8145 [2026-10-20 20:54:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8480 [2026-10-20 20:54:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7613 [2026-10-20 20:54:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3657 [2026-10-20 20:54:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3548 [2026-10-20 20:55:00][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,579 | MFU: 1.8% [2026-10-20 20:55:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6224 [2026-10-20 20:55:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5033 [2026-10-20 20:55:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9229 [2026-10-20 20:55:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3761 [2026-10-20 20:55:05][INFO][training.pretrain_engine] [Ph2|Step 657] loss=5.5441 | ppl=255.73 | aux=0.02508 | lr=3.00e-04 | gnorm=1.156 | tok/s=29,239 | MFU=7.0% | step=143447ms | tokens=0.696B | mem=4.2/32.7GB | cost=$345.29/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 12:48 (135.9h) | Phase3: SKIPPED | SFT: Oct 26 17:33 (140.6h) | DPO (Final): Oct 26 18:07 (141.2h) [2026-10-20 20:55:05][INFO][training.pretrain_engine] Starting global step 658 (accumulating 32 micro-batches)... [2026-10-20 20:55:05][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 20:55:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9363 [2026-10-20 20:55:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4971 [2026-10-20 20:55:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4611 [2026-10-20 20:55:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5648 [2026-10-20 20:55:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1679 [2026-10-20 20:55:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4493 [2026-10-20 20:55:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1361 [2026-10-20 20:55:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0885 [2026-10-20 20:55:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2169 [2026-10-20 20:55:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4822 [2026-10-20 20:55:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0086 [2026-10-20 20:55:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5861 [2026-10-20 20:56:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7850 [2026-10-20 20:56:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8222 [2026-10-20 20:56:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3406 [2026-10-20 20:56:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3531 [2026-10-20 20:56:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8678 [2026-10-20 20:56:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5247 [2026-10-20 20:56:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7560 [2026-10-20 20:56:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6408 [2026-10-20 20:56:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9218 [2026-10-20 20:56:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7144 [2026-10-20 20:56:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5856 [2026-10-20 20:56:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7765 [2026-10-20 20:57:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6108 [2026-10-20 20:57:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6768 [2026-10-20 20:57:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7579 [2026-10-20 20:57:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5212 [2026-10-20 20:57:24][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,556 | MFU: 1.8% [2026-10-20 20:57:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5379 [2026-10-20 20:57:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8215 [2026-10-20 20:57:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6481 [2026-10-20 20:57:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5183 [2026-10-20 20:57:28][INFO][training.pretrain_engine] [Ph2|Step 658] loss=5.5881 | ppl=267.23 | aux=0.02509 | lr=3.00e-04 | gnorm=0.984 | tok/s=29,239 | MFU=7.0% | step=143450ms | tokens=0.697B | mem=4.2/32.7GB | cost=$345.81/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 12:50 (135.9h) | Phase3: SKIPPED | SFT: Oct 26 17:35 (140.6h) | DPO (Final): Oct 26 18:09 (141.2h) [2026-10-20 20:57:28][INFO][training.pretrain_engine] Starting global step 659 (accumulating 32 micro-batches)... [2026-10-20 20:57:28][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 20:57:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4923 [2026-10-20 20:57:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3156 [2026-10-20 20:57:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6651 [2026-10-20 20:57:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7862 [2026-10-20 20:58:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9032 [2026-10-20 20:58:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5662 [2026-10-20 20:58:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4643 [2026-10-20 20:58:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.7434 [2026-10-20 20:58:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6739 [2026-10-20 20:58:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0578 [2026-10-20 20:58:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6033 [2026-10-20 20:58:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5845 [2026-10-20 20:58:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4160 [2026-10-20 20:58:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5905 [2026-10-20 20:58:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4570 [2026-10-20 20:58:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9713 [2026-10-20 20:58:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8605 [2026-10-20 20:58:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5486 [2026-10-20 20:58:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1081 [2026-10-20 20:58:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9359 [2026-10-20 20:59:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6571 [2026-10-20 20:59:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5820 [2026-10-20 20:59:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5535 [2026-10-20 20:59:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4725 [2026-10-20 20:59:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5200 [2026-10-20 20:59:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8479 [2026-10-20 20:59:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6809 [2026-10-20 20:59:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6130 [2026-10-20 20:59:47][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,552 | MFU: 1.8% [2026-10-20 20:59:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4676 [2026-10-20 20:59:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6435 [2026-10-20 20:59:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6083 [2026-10-20 20:59:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5180 [2026-10-20 20:59:52][INFO][training.pretrain_engine] [Ph2|Step 659] loss=5.4367 | ppl=229.69 | aux=0.02508 | lr=3.00e-04 | gnorm=1.000 | tok/s=29,238 | MFU=7.0% | step=143452ms | tokens=0.698B | mem=4.2/32.7GB | cost=$346.34/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 12:52 (135.9h) | Phase3: SKIPPED | SFT: Oct 26 17:37 (140.6h) | DPO (Final): Oct 26 18:11 (141.2h) [2026-10-20 20:59:55][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step659 | Tokens: 0.698B | Spent: $346.35 | Remaining: $53.65 [2026-10-20 20:59:55][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step659 [2026-10-20 20:59:55][INFO][training.pretrain_engine] Starting global step 660 (accumulating 32 micro-batches)... [2026-10-20 20:59:55][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 20:59:55][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 20:59:55][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 20:59:55][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 20:59:55][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step659 to Phase-Technologies/forge-500m-pretrain (path: phase2_step659)... [2026-10-20 20:59:55][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 21:00:05][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 21:00:05][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step659 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 21:00:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7934 [2026-10-20 21:00:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0525 [2026-10-20 21:00:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5937 [2026-10-20 21:00:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7262 [2026-10-20 21:00:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2331 [2026-10-20 21:00:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2076 [2026-10-20 21:00:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6890 [2026-10-20 21:00:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7819 [2026-10-20 21:00:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7559 [2026-10-20 21:00:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8474 [2026-10-20 21:00:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5185 [2026-10-20 21:00:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6505 [2026-10-20 21:01:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4036 [2026-10-20 21:01:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1847 [2026-10-20 21:01:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9482 [2026-10-20 21:01:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8884 [2026-10-20 21:01:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0496 [2026-10-20 21:01:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2883 [2026-10-20 21:01:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4297 [2026-10-20 21:01:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6100 [2026-10-20 21:01:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3332 [2026-10-20 21:01:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9607 [2026-10-20 21:01:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6834 [2026-10-20 21:01:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7282 [2026-10-20 21:01:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1252 [2026-10-20 21:01:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6266 [2026-10-20 21:01:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6516 [2026-10-20 21:02:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4229 [2026-10-20 21:02:13][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,569 | MFU: 1.8% [2026-10-20 21:02:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8413 [2026-10-20 21:02:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8561 [2026-10-20 21:02:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8595 [2026-10-20 21:02:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2992 [2026-10-20 21:02:18][INFO][training.pretrain_engine] [Ph2|Step 660] loss=5.4862 | ppl=241.33 | aux=0.02509 | lr=3.00e-04 | gnorm=0.898 | tok/s=29,238 | MFU=7.0% | step=143453ms | tokens=0.699B | mem=4.2/32.7GB | cost=$346.87/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 12:54 (135.9h) | Phase3: SKIPPED | SFT: Oct 26 17:39 (140.6h) | DPO (Final): Oct 26 18:13 (141.2h) [2026-10-20 21:02:18][INFO][training.pretrain_engine] Starting global step 661 (accumulating 32 micro-batches)... [2026-10-20 21:02:18][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 21:02:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6469 [2026-10-20 21:02:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6094 [2026-10-20 21:02:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7527 [2026-10-20 21:02:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3652 [2026-10-20 21:02:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8584 [2026-10-20 21:02:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9262 [2026-10-20 21:02:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1244 [2026-10-20 21:02:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0422 [2026-10-20 21:03:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0846 [2026-10-20 21:03:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1772 [2026-10-20 21:03:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3478 [2026-10-20 21:03:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1481 [2026-10-20 21:03:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2969 [2026-10-20 21:03:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6583 [2026-10-20 21:03:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2076 [2026-10-20 21:03:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5096 [2026-10-20 21:03:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2442 [2026-10-20 21:03:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6742 [2026-10-20 21:03:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1986 [2026-10-20 21:03:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0495 [2026-10-20 21:04:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6160 [2026-10-20 21:04:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7629 [2026-10-20 21:04:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5337 [2026-10-20 21:04:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5966 [2026-10-20 21:04:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4194 [2026-10-20 21:04:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2555 [2026-10-20 21:04:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7396 [2026-10-20 21:04:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0866 [2026-10-20 21:04:37][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,562 | MFU: 1.8% [2026-10-20 21:04:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4151 [2026-10-20 21:04:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8453 [2026-10-20 21:04:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7163 [2026-10-20 21:04:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2791 [2026-10-20 21:04:41][INFO][training.pretrain_engine] [Ph2|Step 661] loss=5.5650 | ppl=261.13 | aux=0.02509 | lr=3.00e-04 | gnorm=0.832 | tok/s=29,239 | MFU=7.0% | step=143450ms | tokens=0.700B | mem=4.2/32.7GB | cost=$347.39/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 12:55 (135.9h) | Phase3: SKIPPED | SFT: Oct 26 17:40 (140.6h) | DPO (Final): Oct 26 18:14 (141.2h) [2026-10-20 21:04:41][INFO][training.pretrain_engine] Starting global step 662 (accumulating 32 micro-batches)... [2026-10-20 21:04:41][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 21:04:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4838 [2026-10-20 21:04:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4952 [2026-10-20 21:04:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5310 [2026-10-20 21:04:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4857 [2026-10-20 21:05:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7730 [2026-10-20 21:05:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2782 [2026-10-20 21:05:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0582 [2026-10-20 21:05:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5820 [2026-10-20 21:05:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1672 [2026-10-20 21:05:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8967 [2026-10-20 21:05:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5824 [2026-10-20 21:05:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9743 [2026-10-20 21:05:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4049 [2026-10-20 21:05:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8669 [2026-10-20 21:05:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9564 [2026-10-20 21:05:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3226 [2026-10-20 21:06:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0133 [2026-10-20 21:06:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7299 [2026-10-20 21:06:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4912 [2026-10-20 21:06:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9854 [2026-10-20 21:06:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3620 [2026-10-20 21:06:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5283 [2026-10-20 21:06:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2864 [2026-10-20 21:06:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5189 [2026-10-20 21:06:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7047 [2026-10-20 21:06:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4727 [2026-10-20 21:06:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0139 [2026-10-20 21:06:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8345 [2026-10-20 21:07:00][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,558 | MFU: 1.8% [2026-10-20 21:07:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5755 [2026-10-20 21:07:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8115 [2026-10-20 21:07:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5313 [2026-10-20 21:07:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5494 [2026-10-20 21:07:05][INFO][training.pretrain_engine] [Ph2|Step 662] loss=5.5938 | ppl=268.75 | aux=0.02507 | lr=3.00e-04 | gnorm=0.758 | tok/s=29,238 | MFU=7.0% | step=143455ms | tokens=0.701B | mem=4.2/32.7GB | cost=$347.92/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 12:57 (135.8h) | Phase3: SKIPPED | SFT: Oct 26 17:42 (140.6h) | DPO (Final): Oct 26 18:17 (141.2h) [2026-10-20 21:07:05][INFO][training.pretrain_engine] Starting global step 663 (accumulating 32 micro-batches)... [2026-10-20 21:07:05][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 21:07:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7302 [2026-10-20 21:07:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0661 [2026-10-20 21:07:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7592 [2026-10-20 21:07:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1503 [2026-10-20 21:07:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9918 [2026-10-20 21:07:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6437 [2026-10-20 21:07:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6301 [2026-10-20 21:07:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8088 [2026-10-20 21:07:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8907 [2026-10-20 21:07:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1075 [2026-10-20 21:07:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9516 [2026-10-20 21:07:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6430 [2026-10-20 21:08:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3161 [2026-10-20 21:08:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5230 [2026-10-20 21:08:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6685 [2026-10-20 21:08:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3371 [2026-10-20 21:08:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9020 [2026-10-20 21:08:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9675 [2026-10-20 21:08:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8054 [2026-10-20 21:08:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3971 [2026-10-20 21:08:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4063 [2026-10-20 21:08:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6803 [2026-10-20 21:08:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3580 [2026-10-20 21:08:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9098 [2026-10-20 21:09:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3687 [2026-10-20 21:09:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3965 [2026-10-20 21:09:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3955 [2026-10-20 21:09:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3015 [2026-10-20 21:09:24][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,546 | MFU: 1.8% [2026-10-20 21:09:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6057 [2026-10-20 21:09:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9265 [2026-10-20 21:09:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6421 [2026-10-20 21:09:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8413 [2026-10-20 21:09:28][INFO][training.pretrain_engine] [Ph2|Step 663] loss=5.5272 | ppl=251.43 | aux=0.02508 | lr=3.00e-04 | gnorm=0.758 | tok/s=29,236 | MFU=7.0% | step=143461ms | tokens=0.703B | mem=4.2/32.7GB | cost=$348.44/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 12:59 (135.8h) | Phase3: SKIPPED | SFT: Oct 26 17:44 (140.6h) | DPO (Final): Oct 26 18:19 (141.2h) [2026-10-20 21:09:28][INFO][training.pretrain_engine] Starting global step 664 (accumulating 32 micro-batches)... [2026-10-20 21:09:28][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 21:09:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8707 [2026-10-20 21:09:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1907 [2026-10-20 21:09:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6503 [2026-10-20 21:09:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6016 [2026-10-20 21:10:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6141 [2026-10-20 21:10:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5077 [2026-10-20 21:10:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7236 [2026-10-20 21:10:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5700 [2026-10-20 21:10:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2601 [2026-10-20 21:10:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5970 [2026-10-20 21:10:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8746 [2026-10-20 21:10:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6477 [2026-10-20 21:10:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3160 [2026-10-20 21:10:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2416 [2026-10-20 21:10:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4422 [2026-10-20 21:10:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4989 [2026-10-20 21:10:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5568 [2026-10-20 21:10:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3342 [2026-10-20 21:10:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4612 [2026-10-20 21:10:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6546 [2026-10-20 21:11:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1144 [2026-10-20 21:11:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9780 [2026-10-20 21:11:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6459 [2026-10-20 21:11:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9354 [2026-10-20 21:11:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7235 [2026-10-20 21:11:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6034 [2026-10-20 21:11:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5895 [2026-10-20 21:11:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6019 [2026-10-20 21:11:47][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,540 | MFU: 1.8% [2026-10-20 21:11:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7166 [2026-10-20 21:11:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0788 [2026-10-20 21:11:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0526 [2026-10-20 21:11:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3384 [2026-10-20 21:11:52][INFO][training.pretrain_engine] [Ph2|Step 664] loss=5.6446 | ppl=282.76 | aux=0.02508 | lr=3.00e-04 | gnorm=0.746 | tok/s=29,236 | MFU=7.0% | step=143463ms | tokens=0.704B | mem=4.2/32.7GB | cost=$348.97/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 13:01 (135.8h) | Phase3: SKIPPED | SFT: Oct 26 17:46 (140.6h) | DPO (Final): Oct 26 18:21 (141.2h) [2026-10-20 21:11:52][INFO][training.pretrain_engine] Starting global step 665 (accumulating 32 micro-batches)... [2026-10-20 21:11:52][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 21:12:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9663 [2026-10-20 21:12:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1652 [2026-10-20 21:12:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8235 [2026-10-20 21:12:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7424 [2026-10-20 21:12:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4749 [2026-10-20 21:12:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9384 [2026-10-20 21:12:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6322 [2026-10-20 21:12:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4946 [2026-10-20 21:12:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0005 [2026-10-20 21:12:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8399 [2026-10-20 21:12:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7262 [2026-10-20 21:12:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6904 [2026-10-20 21:13:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1009 [2026-10-20 21:13:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7154 [2026-10-20 21:13:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4424 [2026-10-20 21:13:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1529 [2026-10-20 21:13:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3089 [2026-10-20 21:13:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1638 [2026-10-20 21:13:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7122 [2026-10-20 21:13:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7215 [2026-10-20 21:13:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3797 [2026-10-20 21:13:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3974 [2026-10-20 21:13:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4507 [2026-10-20 21:13:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8333 [2026-10-20 21:13:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5700 [2026-10-20 21:13:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3428 [2026-10-20 21:13:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1611 [2026-10-20 21:13:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7828 [2026-10-20 21:14:10][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,570 | MFU: 1.8% [2026-10-20 21:14:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7720 [2026-10-20 21:14:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6579 [2026-10-20 21:14:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0583 [2026-10-20 21:14:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5003 [2026-10-20 21:14:15][INFO][training.pretrain_engine] [Ph2|Step 665] loss=5.5794 | ppl=264.91 | aux=0.02507 | lr=3.00e-04 | gnorm=0.676 | tok/s=29,237 | MFU=7.0% | step=143460ms | tokens=0.705B | mem=4.2/32.7GB | cost=$349.49/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 13:03 (135.8h) | Phase3: SKIPPED | SFT: Oct 26 17:48 (140.6h) | DPO (Final): Oct 26 18:22 (141.1h) [2026-10-20 21:14:15][INFO][training.pretrain_engine] Starting global step 666 (accumulating 32 micro-batches)... [2026-10-20 21:14:15][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 21:14:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4209 [2026-10-20 21:14:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8379 [2026-10-20 21:14:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4819 [2026-10-20 21:14:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9778 [2026-10-20 21:14:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2020 [2026-10-20 21:14:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0913 [2026-10-20 21:14:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8209 [2026-10-20 21:14:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6525 [2026-10-20 21:15:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2918 [2026-10-20 21:15:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1697 [2026-10-20 21:15:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3597 [2026-10-20 21:15:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0646 [2026-10-20 21:15:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7550 [2026-10-20 21:15:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6015 [2026-10-20 21:15:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5763 [2026-10-20 21:15:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5601 [2026-10-20 21:15:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8257 [2026-10-20 21:15:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9593 [2026-10-20 21:15:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3910 [2026-10-20 21:15:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4822 [2026-10-20 21:16:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6541 [2026-10-20 21:16:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5408 [2026-10-20 21:16:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0603 [2026-10-20 21:16:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6096 [2026-10-20 21:16:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1924 [2026-10-20 21:16:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8761 [2026-10-20 21:16:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4017 [2026-10-20 21:16:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2767 [2026-10-20 21:16:33][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,596 | MFU: 1.8% [2026-10-20 21:16:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3175 [2026-10-20 21:16:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3459 [2026-10-20 21:16:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0267 [2026-10-20 21:16:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5300 [2026-10-20 21:16:38][INFO][training.pretrain_engine] [Ph2|Step 666] loss=5.3647 | ppl=213.73 | aux=0.02508 | lr=3.00e-04 | gnorm=0.688 | tok/s=29,236 | MFU=7.0% | step=143461ms | tokens=0.706B | mem=4.2/32.7GB | cost=$350.01/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 13:05 (135.8h) | Phase3: SKIPPED | SFT: Oct 26 17:50 (140.6h) | DPO (Final): Oct 26 18:24 (141.1h) [2026-10-20 21:16:38][INFO][training.pretrain_engine] Starting global step 667 (accumulating 32 micro-batches)... [2026-10-20 21:16:38][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 21:16:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1643 [2026-10-20 21:16:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6881 [2026-10-20 21:16:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0253 [2026-10-20 21:16:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8226 [2026-10-20 21:17:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6039 [2026-10-20 21:17:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3027 [2026-10-20 21:17:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2973 [2026-10-20 21:17:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9301 [2026-10-20 21:17:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6278 [2026-10-20 21:17:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8215 [2026-10-20 21:17:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4120 [2026-10-20 21:17:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5464 [2026-10-20 21:17:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4574 [2026-10-20 21:17:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0784 [2026-10-20 21:17:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5874 [2026-10-20 21:17:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1223 [2026-10-20 21:18:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4874 [2026-10-20 21:18:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3654 [2026-10-20 21:18:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8419 [2026-10-20 21:18:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4382 [2026-10-20 21:18:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0725 [2026-10-20 21:18:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7075 [2026-10-20 21:18:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4224 [2026-10-20 21:18:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9265 [2026-10-20 21:18:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9007 [2026-10-20 21:18:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7265 [2026-10-20 21:18:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4290 [2026-10-20 21:18:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7429 [2026-10-20 21:18:56][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,567 | MFU: 1.8% [2026-10-20 21:19:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6378 [2026-10-20 21:19:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4152 [2026-10-20 21:19:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5695 [2026-10-20 21:19:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7413 [2026-10-20 21:19:01][INFO][training.pretrain_engine] [Ph2|Step 667] loss=5.5839 | ppl=266.10 | aux=0.02508 | lr=3.00e-04 | gnorm=0.801 | tok/s=29,237 | MFU=7.0% | step=143459ms | tokens=0.707B | mem=4.2/32.7GB | cost=$350.54/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 13:06 (135.8h) | Phase3: SKIPPED | SFT: Oct 26 17:51 (140.5h) | DPO (Final): Oct 26 18:26 (141.1h) [2026-10-20 21:19:01][INFO][training.pretrain_engine] Starting global step 668 (accumulating 32 micro-batches)... [2026-10-20 21:19:01][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 21:19:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4130 [2026-10-20 21:19:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7730 [2026-10-20 21:19:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5189 [2026-10-20 21:19:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1990 [2026-10-20 21:19:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2906 [2026-10-20 21:19:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2655 [2026-10-20 21:19:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0953 [2026-10-20 21:19:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9315 [2026-10-20 21:19:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4647 [2026-10-20 21:19:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4313 [2026-10-20 21:19:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6789 [2026-10-20 21:19:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6378 [2026-10-20 21:20:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9556 [2026-10-20 21:20:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6160 [2026-10-20 21:20:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1575 [2026-10-20 21:20:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1319 [2026-10-20 21:20:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2464 [2026-10-20 21:20:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9306 [2026-10-20 21:20:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4811 [2026-10-20 21:20:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5622 [2026-10-20 21:20:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3190 [2026-10-20 21:20:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7996 [2026-10-20 21:20:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5912 [2026-10-20 21:20:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0431 [2026-10-20 21:21:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7948 [2026-10-20 21:21:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1795 [2026-10-20 21:21:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7396 [2026-10-20 21:21:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2109 [2026-10-20 21:21:19][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,566 | MFU: 1.8% [2026-10-20 21:21:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3072 [2026-10-20 21:21:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6043 [2026-10-20 21:21:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9387 [2026-10-20 21:21:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9570 [2026-10-20 21:21:24][INFO][training.pretrain_engine] [Ph2|Step 668] loss=5.6195 | ppl=275.74 | aux=0.02509 | lr=3.00e-04 | gnorm=0.688 | tok/s=29,237 | MFU=7.0% | step=143456ms | tokens=0.708B | mem=4.2/32.7GB | cost=$351.06/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 13:08 (135.8h) | Phase3: SKIPPED | SFT: Oct 26 17:53 (140.5h) | DPO (Final): Oct 26 18:27 (141.1h) [2026-10-20 21:21:27][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step668 | Tokens: 0.708B | Spent: $351.07 | Remaining: $48.93 [2026-10-20 21:21:27][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step668 [2026-10-20 21:21:27][INFO][training.pretrain_engine] Starting global step 669 (accumulating 32 micro-batches)... [2026-10-20 21:21:27][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 21:21:27][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 21:21:27][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 21:21:27][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 21:21:27][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step668 to Phase-Technologies/forge-500m-pretrain (path: phase2_step668)... [2026-10-20 21:21:28][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 21:21:38][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 21:21:38][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step668 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 21:21:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5955 [2026-10-20 21:21:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5653 [2026-10-20 21:21:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2472 [2026-10-20 21:21:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8705 [2026-10-20 21:22:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6218 [2026-10-20 21:22:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4495 [2026-10-20 21:22:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2922 [2026-10-20 21:22:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8856 [2026-10-20 21:22:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1874 [2026-10-20 21:22:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2432 [2026-10-20 21:22:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8882 [2026-10-20 21:22:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3517 [2026-10-20 21:22:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6492 [2026-10-20 21:22:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2766 [2026-10-20 21:22:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2230 [2026-10-20 21:22:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2243 [2026-10-20 21:22:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3616 [2026-10-20 21:22:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6949 [2026-10-20 21:22:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8361 [2026-10-20 21:22:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4338 [2026-10-20 21:23:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8550 [2026-10-20 21:23:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3283 [2026-10-20 21:23:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9953 [2026-10-20 21:23:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5070 [2026-10-20 21:23:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8085 [2026-10-20 21:23:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3443 [2026-10-20 21:23:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9312 [2026-10-20 21:23:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9304 [2026-10-20 21:23:46][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,552 | MFU: 1.8% [2026-10-20 21:23:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4982 [2026-10-20 21:23:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0988 [2026-10-20 21:23:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4100 [2026-10-20 21:23:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4101 [2026-10-20 21:23:51][INFO][training.pretrain_engine] [Ph2|Step 669] loss=5.5135 | ppl=248.03 | aux=0.02509 | lr=3.00e-04 | gnorm=0.699 | tok/s=29,237 | MFU=7.0% | step=143457ms | tokens=0.709B | mem=4.2/32.7GB | cost=$351.59/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 13:10 (135.8h) | Phase3: SKIPPED | SFT: Oct 26 17:55 (140.5h) | DPO (Final): Oct 26 18:29 (141.1h) [2026-10-20 21:23:51][INFO][training.pretrain_engine] Starting global step 670 (accumulating 32 micro-batches)... [2026-10-20 21:23:51][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 21:24:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2380 [2026-10-20 21:24:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4815 [2026-10-20 21:24:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8444 [2026-10-20 21:24:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9358 [2026-10-20 21:24:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2171 [2026-10-20 21:24:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1708 [2026-10-20 21:24:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6111 [2026-10-20 21:24:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4341 [2026-10-20 21:24:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3716 [2026-10-20 21:24:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0326 [2026-10-20 21:24:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7575 [2026-10-20 21:24:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5438 [2026-10-20 21:25:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6723 [2026-10-20 21:25:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6102 [2026-10-20 21:25:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0022 [2026-10-20 21:25:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2002 [2026-10-20 21:25:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7200 [2026-10-20 21:25:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7203 [2026-10-20 21:25:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6152 [2026-10-20 21:25:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3944 [2026-10-20 21:25:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8210 [2026-10-20 21:25:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3119 [2026-10-20 21:25:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4344 [2026-10-20 21:25:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5204 [2026-10-20 21:25:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9405 [2026-10-20 21:25:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9239 [2026-10-20 21:25:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5066 [2026-10-20 21:25:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2360 [2026-10-20 21:26:09][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,569 | MFU: 1.8% [2026-10-20 21:26:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4647 [2026-10-20 21:26:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2947 [2026-10-20 21:26:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6434 [2026-10-20 21:26:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2464 [2026-10-20 21:26:14][INFO][training.pretrain_engine] [Ph2|Step 670] loss=5.4698 | ppl=237.41 | aux=0.02510 | lr=3.00e-04 | gnorm=0.883 | tok/s=29,236 | MFU=7.0% | step=143463ms | tokens=0.710B | mem=4.2/32.7GB | cost=$352.12/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 13:12 (135.8h) | Phase3: SKIPPED | SFT: Oct 26 17:57 (140.5h) | DPO (Final): Oct 26 18:31 (141.1h) [2026-10-20 21:26:14][INFO][training.pretrain_engine] Starting global step 671 (accumulating 32 micro-batches)... [2026-10-20 21:26:14][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 21:26:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5569 [2026-10-20 21:26:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6944 [2026-10-20 21:26:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3957 [2026-10-20 21:26:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4855 [2026-10-20 21:26:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0075 [2026-10-20 21:26:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0697 [2026-10-20 21:26:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8785 [2026-10-20 21:26:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8289 [2026-10-20 21:27:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0442 [2026-10-20 21:27:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7662 [2026-10-20 21:27:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0820 [2026-10-20 21:27:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3195 [2026-10-20 21:27:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4946 [2026-10-20 21:27:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4709 [2026-10-20 21:27:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7880 [2026-10-20 21:27:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2047 [2026-10-20 21:27:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5296 [2026-10-20 21:27:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2451 [2026-10-20 21:27:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6779 [2026-10-20 21:27:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3360 [2026-10-20 21:28:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5409 [2026-10-20 21:28:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1674 [2026-10-20 21:28:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8343 [2026-10-20 21:28:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2481 [2026-10-20 21:28:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2041 [2026-10-20 21:28:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2959 [2026-10-20 21:28:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0227 [2026-10-20 21:28:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8464 [2026-10-20 21:28:33][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,536 | MFU: 1.8% [2026-10-20 21:28:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4535 [2026-10-20 21:28:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6450 [2026-10-20 21:28:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5771 [2026-10-20 21:28:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5372 [2026-10-20 21:28:38][INFO][training.pretrain_engine] [Ph2|Step 671] loss=5.5455 | ppl=256.08 | aux=0.02509 | lr=3.00e-04 | gnorm=0.973 | tok/s=29,235 | MFU=7.0% | step=143468ms | tokens=0.711B | mem=4.2/32.7GB | cost=$352.65/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 13:14 (135.8h) | Phase3: SKIPPED | SFT: Oct 26 17:59 (140.5h) | DPO (Final): Oct 26 18:33 (141.1h) [2026-10-20 21:28:38][INFO][training.pretrain_engine] Starting global step 672 (accumulating 32 micro-batches)... [2026-10-20 21:28:38][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 21:28:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8884 [2026-10-20 21:28:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0286 [2026-10-20 21:28:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5354 [2026-10-20 21:28:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6259 [2026-10-20 21:29:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0276 [2026-10-20 21:29:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0222 [2026-10-20 21:29:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6216 [2026-10-20 21:29:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2074 [2026-10-20 21:29:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3508 [2026-10-20 21:29:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2909 [2026-10-20 21:29:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1477 [2026-10-20 21:29:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9154 [2026-10-20 21:29:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4995 [2026-10-20 21:29:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4734 [2026-10-20 21:29:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4428 [2026-10-20 21:29:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5574 [2026-10-20 21:30:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2704 [2026-10-20 21:30:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6148 [2026-10-20 21:30:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6942 [2026-10-20 21:30:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8232 [2026-10-20 21:30:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4417 [2026-10-20 21:30:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8413 [2026-10-20 21:30:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7473 [2026-10-20 21:30:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3572 [2026-10-20 21:30:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8101 [2026-10-20 21:30:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0743 [2026-10-20 21:30:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1912 [2026-10-20 21:30:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4346 [2026-10-20 21:30:57][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,546 | MFU: 1.8% [2026-10-20 21:31:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8810 [2026-10-20 21:31:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5523 [2026-10-20 21:31:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2662 [2026-10-20 21:31:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7901 [2026-10-20 21:31:01][INFO][training.pretrain_engine] [Ph2|Step 672] loss=5.5445 | ppl=255.83 | aux=0.02509 | lr=3.00e-04 | gnorm=1.062 | tok/s=29,235 | MFU=7.0% | step=143469ms | tokens=0.712B | mem=4.2/32.7GB | cost=$353.17/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 13:16 (135.8h) | Phase3: SKIPPED | SFT: Oct 26 18:01 (140.5h) | DPO (Final): Oct 26 18:35 (141.1h) [2026-10-20 21:31:01][INFO][training.pretrain_engine] Starting global step 673 (accumulating 32 micro-batches)... [2026-10-20 21:31:01][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 21:31:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5413 [2026-10-20 21:31:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1902 [2026-10-20 21:31:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5458 [2026-10-20 21:31:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3501 [2026-10-20 21:31:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5234 [2026-10-20 21:31:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1242 [2026-10-20 21:31:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8532 [2026-10-20 21:31:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1502 [2026-10-20 21:31:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6273 [2026-10-20 21:31:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3149 [2026-10-20 21:31:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5144 [2026-10-20 21:31:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2801 [2026-10-20 21:32:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6113 [2026-10-20 21:32:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9474 [2026-10-20 21:32:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8502 [2026-10-20 21:32:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4516 [2026-10-20 21:32:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5560 [2026-10-20 21:32:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6191 [2026-10-20 21:32:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1932 [2026-10-20 21:32:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0438 [2026-10-20 21:32:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3280 [2026-10-20 21:32:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7360 [2026-10-20 21:32:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7990 [2026-10-20 21:32:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3108 [2026-10-20 21:33:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5163 [2026-10-20 21:33:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6765 [2026-10-20 21:33:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3881 [2026-10-20 21:33:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3455 [2026-10-20 21:33:20][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,568 | MFU: 1.8% [2026-10-20 21:33:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6484 [2026-10-20 21:33:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8115 [2026-10-20 21:33:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5724 [2026-10-20 21:33:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7451 [2026-10-20 21:33:25][INFO][training.pretrain_engine] [Ph2|Step 673] loss=5.6446 | ppl=282.76 | aux=0.02509 | lr=3.00e-04 | gnorm=1.031 | tok/s=29,235 | MFU=7.0% | step=143471ms | tokens=0.713B | mem=4.2/32.7GB | cost=$353.69/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 13:18 (135.8h) | Phase3: SKIPPED | SFT: Oct 26 18:03 (140.5h) | DPO (Final): Oct 26 18:37 (141.1h) [2026-10-20 21:33:25][INFO][training.pretrain_engine] Starting global step 674 (accumulating 32 micro-batches)... [2026-10-20 21:33:25][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 21:33:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4991 [2026-10-20 21:33:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6565 [2026-10-20 21:33:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6039 [2026-10-20 21:33:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3608 [2026-10-20 21:34:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5021 [2026-10-20 21:34:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9371 [2026-10-20 21:34:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3841 [2026-10-20 21:34:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2184 [2026-10-20 21:34:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3131 [2026-10-20 21:34:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4221 [2026-10-20 21:34:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9165 [2026-10-20 21:34:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4106 [2026-10-20 21:34:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6144 [2026-10-20 21:34:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4466 [2026-10-20 21:34:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4590 [2026-10-20 21:34:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5311 [2026-10-20 21:34:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5371 [2026-10-20 21:34:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4414 [2026-10-20 21:34:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6739 [2026-10-20 21:34:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1396 [2026-10-20 21:35:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5089 [2026-10-20 21:35:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6964 [2026-10-20 21:35:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7455 [2026-10-20 21:35:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8206 [2026-10-20 21:35:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5879 [2026-10-20 21:35:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6231 [2026-10-20 21:35:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8798 [2026-10-20 21:35:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6419 [2026-10-20 21:35:43][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,578 | MFU: 1.8% [2026-10-20 21:35:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2920 [2026-10-20 21:35:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0389 [2026-10-20 21:35:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7389 [2026-10-20 21:35:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1915 [2026-10-20 21:35:49][INFO][training.pretrain_engine] [Ph2|Step 674] loss=5.5517 | ppl=257.68 | aux=0.02508 | lr=3.00e-04 | gnorm=1.203 | tok/s=29,232 | MFU=7.0% | step=143482ms | tokens=0.714B | mem=4.2/32.7GB | cost=$354.22/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 13:20 (135.8h) | Phase3: SKIPPED | SFT: Oct 26 18:05 (140.5h) | DPO (Final): Oct 26 18:40 (141.1h) [2026-10-20 21:35:49][INFO][training.pretrain_engine] Starting global step 675 (accumulating 32 micro-batches)... [2026-10-20 21:35:49][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 21:36:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7398 [2026-10-20 21:36:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4072 [2026-10-20 21:36:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3975 [2026-10-20 21:36:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3788 [2026-10-20 21:36:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0021 [2026-10-20 21:36:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0878 [2026-10-20 21:36:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2181 [2026-10-20 21:36:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1175 [2026-10-20 21:36:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5633 [2026-10-20 21:36:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5989 [2026-10-20 21:36:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6669 [2026-10-20 21:36:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8695 [2026-10-20 21:37:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5553 [2026-10-20 21:37:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6565 [2026-10-20 21:37:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7249 [2026-10-20 21:37:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5941 [2026-10-20 21:37:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0520 [2026-10-20 21:37:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0477 [2026-10-20 21:37:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7818 [2026-10-20 21:37:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5227 [2026-10-20 21:37:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2472 [2026-10-20 21:37:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6144 [2026-10-20 21:37:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4243 [2026-10-20 21:37:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0273 [2026-10-20 21:37:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3564 [2026-10-20 21:37:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2676 [2026-10-20 21:37:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0640 [2026-10-20 21:37:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6643 [2026-10-20 21:38:08][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,564 | MFU: 1.8% [2026-10-20 21:38:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6930 [2026-10-20 21:38:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6100 [2026-10-20 21:38:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5209 [2026-10-20 21:38:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6037 [2026-10-20 21:38:13][INFO][training.pretrain_engine] [Ph2|Step 675] loss=5.5833 | ppl=265.95 | aux=0.02510 | lr=3.00e-04 | gnorm=0.738 | tok/s=29,232 | MFU=7.0% | step=143485ms | tokens=0.715B | mem=4.2/32.7GB | cost=$354.75/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 13:22 (135.7h) | Phase3: SKIPPED | SFT: Oct 26 18:07 (140.5h) | DPO (Final): Oct 26 18:42 (141.1h) [2026-10-20 21:38:13][INFO][training.pretrain_engine] Starting global step 676 (accumulating 32 micro-batches)... [2026-10-20 21:38:13][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 21:38:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3324 [2026-10-20 21:38:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4606 [2026-10-20 21:38:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2066 [2026-10-20 21:38:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8049 [2026-10-20 21:38:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3785 [2026-10-20 21:38:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2925 [2026-10-20 21:38:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3359 [2026-10-20 21:38:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5878 [2026-10-20 21:39:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1594 [2026-10-20 21:39:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2028 [2026-10-20 21:39:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7931 [2026-10-20 21:39:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2545 [2026-10-20 21:39:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2420 [2026-10-20 21:39:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9279 [2026-10-20 21:39:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1063 [2026-10-20 21:39:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2174 [2026-10-20 21:39:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8139 [2026-10-20 21:39:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5725 [2026-10-20 21:39:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2194 [2026-10-20 21:39:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5565 [2026-10-20 21:40:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9023 [2026-10-20 21:40:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3641 [2026-10-20 21:40:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5581 [2026-10-20 21:40:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0303 [2026-10-20 21:40:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9153 [2026-10-20 21:40:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6175 [2026-10-20 21:40:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6497 [2026-10-20 21:40:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3460 [2026-10-20 21:40:32][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,527 | MFU: 1.8% [2026-10-20 21:40:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0828 [2026-10-20 21:40:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4795 [2026-10-20 21:40:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6242 [2026-10-20 21:40:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9622 [2026-10-20 21:40:36][INFO][training.pretrain_engine] [Ph2|Step 676] loss=5.5292 | ppl=251.95 | aux=0.02508 | lr=3.00e-04 | gnorm=1.031 | tok/s=29,231 | MFU=7.0% | step=143490ms | tokens=0.716B | mem=4.2/32.7GB | cost=$355.27/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 13:24 (135.7h) | Phase3: SKIPPED | SFT: Oct 26 18:10 (140.5h) | DPO (Final): Oct 26 18:44 (141.1h) [2026-10-20 21:40:36][INFO][training.pretrain_engine] Starting global step 677 (accumulating 32 micro-batches)... [2026-10-20 21:40:36][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 21:40:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1494 [2026-10-20 21:40:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1830 [2026-10-20 21:40:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8706 [2026-10-20 21:40:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3342 [2026-10-20 21:41:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6220 [2026-10-20 21:41:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5891 [2026-10-20 21:41:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0452 [2026-10-20 21:41:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6737 [2026-10-20 21:41:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6774 [2026-10-20 21:41:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1032 [2026-10-20 21:41:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8643 [2026-10-20 21:41:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1349 [2026-10-20 21:41:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7058 [2026-10-20 21:41:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3645 [2026-10-20 21:41:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7451 [2026-10-20 21:41:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5178 [2026-10-20 21:42:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5122 [2026-10-20 21:42:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4355 [2026-10-20 21:42:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0675 [2026-10-20 21:42:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5232 [2026-10-20 21:42:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5064 [2026-10-20 21:42:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8299 [2026-10-20 21:42:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3484 [2026-10-20 21:42:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9687 [2026-10-20 21:42:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6576 [2026-10-20 21:42:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2747 [2026-10-20 21:42:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5055 [2026-10-20 21:42:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5010 [2026-10-20 21:42:56][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,523 | MFU: 1.8% [2026-10-20 21:43:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4787 [2026-10-20 21:43:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3744 [2026-10-20 21:43:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8031 [2026-10-20 21:43:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3378 [2026-10-20 21:43:01][INFO][training.pretrain_engine] [Ph2|Step 677] loss=5.6194 | ppl=275.74 | aux=0.02508 | lr=3.00e-04 | gnorm=0.875 | tok/s=29,228 | MFU=7.0% | step=143502ms | tokens=0.717B | mem=4.2/32.7GB | cost=$355.80/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 13:27 (135.7h) | Phase3: SKIPPED | SFT: Oct 26 18:12 (140.5h) | DPO (Final): Oct 26 18:46 (141.1h) [2026-10-20 21:43:04][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step677 | Tokens: 0.717B | Spent: $355.81 | Remaining: $44.19 [2026-10-20 21:43:04][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step677 [2026-10-20 21:43:04][INFO][training.pretrain_engine] Starting global step 678 (accumulating 32 micro-batches)... [2026-10-20 21:43:04][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 21:43:04][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 21:43:04][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 21:43:04][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 21:43:04][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step677 to Phase-Technologies/forge-500m-pretrain (path: phase2_step677)... [2026-10-20 21:43:04][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 21:43:14][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 21:43:14][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step677 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 21:43:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4834 [2026-10-20 21:43:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8708 [2026-10-20 21:43:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6241 [2026-10-20 21:43:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3388 [2026-10-20 21:43:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8628 [2026-10-20 21:43:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7003 [2026-10-20 21:43:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0384 [2026-10-20 21:43:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6009 [2026-10-20 21:43:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2566 [2026-10-20 21:43:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5843 [2026-10-20 21:43:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6358 [2026-10-20 21:43:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1197 [2026-10-20 21:44:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7142 [2026-10-20 21:44:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4522 [2026-10-20 21:44:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3855 [2026-10-20 21:44:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4359 [2026-10-20 21:44:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2788 [2026-10-20 21:44:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6402 [2026-10-20 21:44:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5585 [2026-10-20 21:44:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3815 [2026-10-20 21:44:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3254 [2026-10-20 21:44:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2433 [2026-10-20 21:44:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7477 [2026-10-20 21:44:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9686 [2026-10-20 21:45:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0191 [2026-10-20 21:45:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9727 [2026-10-20 21:45:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4068 [2026-10-20 21:45:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5703 [2026-10-20 21:45:23][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,559 | MFU: 1.8% [2026-10-20 21:45:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3185 [2026-10-20 21:45:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2601 [2026-10-20 21:45:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6329 [2026-10-20 21:45:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3473 [2026-10-20 21:45:27][INFO][training.pretrain_engine] [Ph2|Step 678] loss=5.5017 | ppl=245.10 | aux=0.02508 | lr=3.00e-04 | gnorm=0.672 | tok/s=29,228 | MFU=7.0% | step=143501ms | tokens=0.718B | mem=4.2/32.7GB | cost=$356.34/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 13:29 (135.7h) | Phase3: SKIPPED | SFT: Oct 26 18:14 (140.5h) | DPO (Final): Oct 26 18:48 (141.1h) [2026-10-20 21:45:27][INFO][training.pretrain_engine] Starting global step 679 (accumulating 32 micro-batches)... [2026-10-20 21:45:27][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 21:45:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5740 [2026-10-20 21:45:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3104 [2026-10-20 21:45:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5971 [2026-10-20 21:45:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6300 [2026-10-20 21:46:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4691 [2026-10-20 21:46:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6382 [2026-10-20 21:46:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5210 [2026-10-20 21:46:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2522 [2026-10-20 21:46:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0751 [2026-10-20 21:46:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7803 [2026-10-20 21:46:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6807 [2026-10-20 21:46:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9683 [2026-10-20 21:46:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3725 [2026-10-20 21:46:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3067 [2026-10-20 21:46:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1897 [2026-10-20 21:46:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6983 [2026-10-20 21:46:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4329 [2026-10-20 21:46:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2643 [2026-10-20 21:46:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6653 [2026-10-20 21:46:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6044 [2026-10-20 21:47:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3988 [2026-10-20 21:47:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7063 [2026-10-20 21:47:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0433 [2026-10-20 21:47:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3504 [2026-10-20 21:47:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5165 [2026-10-20 21:47:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2039 [2026-10-20 21:47:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3953 [2026-10-20 21:47:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6587 [2026-10-20 21:47:46][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,568 | MFU: 1.8% [2026-10-20 21:47:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3089 [2026-10-20 21:47:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9661 [2026-10-20 21:47:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4797 [2026-10-20 21:47:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9227 [2026-10-20 21:47:50][INFO][training.pretrain_engine] [Ph2|Step 679] loss=5.4941 | ppl=243.26 | aux=0.02508 | lr=3.00e-04 | gnorm=0.742 | tok/s=29,231 | MFU=7.0% | step=143490ms | tokens=0.719B | mem=4.2/32.7GB | cost=$356.86/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 13:30 (135.7h) | Phase3: SKIPPED | SFT: Oct 26 18:15 (140.5h) | DPO (Final): Oct 26 18:49 (141.0h) [2026-10-20 21:47:50][INFO][training.pretrain_engine] Starting global step 680 (accumulating 32 micro-batches)... [2026-10-20 21:47:50][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 21:48:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5308 [2026-10-20 21:48:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7796 [2026-10-20 21:48:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6059 [2026-10-20 21:48:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6022 [2026-10-20 21:48:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2890 [2026-10-20 21:48:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1604 [2026-10-20 21:48:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3839 [2026-10-20 21:48:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4280 [2026-10-20 21:48:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6821 [2026-10-20 21:48:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6478 [2026-10-20 21:48:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3380 [2026-10-20 21:48:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0635 [2026-10-20 21:49:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2162 [2026-10-20 21:49:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7048 [2026-10-20 21:49:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2975 [2026-10-20 21:49:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7113 [2026-10-20 21:49:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1305 [2026-10-20 21:49:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9451 [2026-10-20 21:49:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5793 [2026-10-20 21:49:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8738 [2026-10-20 21:49:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7015 [2026-10-20 21:49:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7051 [2026-10-20 21:49:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3250 [2026-10-20 21:49:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5528 [2026-10-20 21:49:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6469 [2026-10-20 21:49:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1146 [2026-10-20 21:49:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4637 [2026-10-20 21:49:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7523 [2026-10-20 21:50:09][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,583 | MFU: 1.8% [2026-10-20 21:50:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4633 [2026-10-20 21:50:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8756 [2026-10-20 21:50:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7672 [2026-10-20 21:50:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4271 [2026-10-20 21:50:14][INFO][training.pretrain_engine] [Ph2|Step 680] loss=5.5676 | ppl=261.80 | aux=0.02508 | lr=3.00e-04 | gnorm=0.707 | tok/s=29,232 | MFU=7.0% | step=143484ms | tokens=0.720B | mem=4.2/32.7GB | cost=$357.38/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 13:31 (135.7h) | Phase3: SKIPPED | SFT: Oct 26 18:16 (140.4h) | DPO (Final): Oct 26 18:51 (141.0h) [2026-10-20 21:50:14][INFO][training.pretrain_engine] Starting global step 681 (accumulating 32 micro-batches)... [2026-10-20 21:50:14][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 21:50:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3448 [2026-10-20 21:50:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4408 [2026-10-20 21:50:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2134 [2026-10-20 21:50:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3710 [2026-10-20 21:50:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2177 [2026-10-20 21:50:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5662 [2026-10-20 21:50:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8492 [2026-10-20 21:50:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7943 [2026-10-20 21:51:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5163 [2026-10-20 21:51:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4644 [2026-10-20 21:51:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5080 [2026-10-20 21:51:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4589 [2026-10-20 21:51:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5322 [2026-10-20 21:51:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0484 [2026-10-20 21:51:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6314 [2026-10-20 21:51:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4727 [2026-10-20 21:51:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2472 [2026-10-20 21:51:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5108 [2026-10-20 21:51:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5778 [2026-10-20 21:51:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0988 [2026-10-20 21:52:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7002 [2026-10-20 21:52:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3137 [2026-10-20 21:52:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0944 [2026-10-20 21:52:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9141 [2026-10-20 21:52:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5000 [2026-10-20 21:52:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4623 [2026-10-20 21:52:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1286 [2026-10-20 21:52:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7496 [2026-10-20 21:52:33][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,542 | MFU: 1.8% [2026-10-20 21:52:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0623 [2026-10-20 21:52:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5896 [2026-10-20 21:52:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4879 [2026-10-20 21:52:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5757 [2026-10-20 21:52:37][INFO][training.pretrain_engine] [Ph2|Step 681] loss=5.5268 | ppl=251.34 | aux=0.02508 | lr=3.00e-04 | gnorm=0.594 | tok/s=29,233 | MFU=7.0% | step=143480ms | tokens=0.721B | mem=4.2/32.7GB | cost=$357.91/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 13:33 (135.7h) | Phase3: SKIPPED | SFT: Oct 26 18:18 (140.4h) | DPO (Final): Oct 26 18:52 (141.0h) [2026-10-20 21:52:37][INFO][training.pretrain_engine] Starting global step 682 (accumulating 32 micro-batches)... [2026-10-20 21:52:37][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 21:52:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5923 [2026-10-20 21:52:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0150 [2026-10-20 21:52:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2411 [2026-10-20 21:52:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6083 [2026-10-20 21:53:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9510 [2026-10-20 21:53:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0305 [2026-10-20 21:53:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7317 [2026-10-20 21:53:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1298 [2026-10-20 21:53:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5675 [2026-10-20 21:53:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7265 [2026-10-20 21:53:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5849 [2026-10-20 21:53:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3273 [2026-10-20 21:53:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7248 [2026-10-20 21:53:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9670 [2026-10-20 21:53:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4867 [2026-10-20 21:53:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6707 [2026-10-20 21:54:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8818 [2026-10-20 21:54:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5857 [2026-10-20 21:54:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9391 [2026-10-20 21:54:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5029 [2026-10-20 21:54:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3485 [2026-10-20 21:54:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9875 [2026-10-20 21:54:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6873 [2026-10-20 21:54:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5891 [2026-10-20 21:54:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5752 [2026-10-20 21:54:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4542 [2026-10-20 21:54:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4938 [2026-10-20 21:54:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5153 [2026-10-20 21:54:56][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,566 | MFU: 1.8% [2026-10-20 21:55:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6298 [2026-10-20 21:55:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9035 [2026-10-20 21:55:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4144 [2026-10-20 21:55:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5615 [2026-10-20 21:55:01][INFO][training.pretrain_engine] [Ph2|Step 682] loss=5.3941 | ppl=220.11 | aux=0.02509 | lr=3.00e-04 | gnorm=0.684 | tok/s=29,233 | MFU=7.0% | step=143477ms | tokens=0.722B | mem=4.2/32.7GB | cost=$358.43/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 13:35 (135.7h) | Phase3: SKIPPED | SFT: Oct 26 18:20 (140.4h) | DPO (Final): Oct 26 18:54 (141.0h) [2026-10-20 21:55:01][INFO][training.pretrain_engine] Starting global step 683 (accumulating 32 micro-batches)... [2026-10-20 21:55:01][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 21:55:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5131 [2026-10-20 21:55:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4379 [2026-10-20 21:55:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7062 [2026-10-20 21:55:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3726 [2026-10-20 21:55:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7830 [2026-10-20 21:55:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3722 [2026-10-20 21:55:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2697 [2026-10-20 21:55:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5032 [2026-10-20 21:55:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4005 [2026-10-20 21:55:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6619 [2026-10-20 21:55:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3583 [2026-10-20 21:55:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3891 [2026-10-20 21:56:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7881 [2026-10-20 21:56:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2079 [2026-10-20 21:56:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5030 [2026-10-20 21:56:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2607 [2026-10-20 21:56:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1996 [2026-10-20 21:56:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5245 [2026-10-20 21:56:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3139 [2026-10-20 21:56:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5583 [2026-10-20 21:56:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5255 [2026-10-20 21:56:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4107 [2026-10-20 21:56:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9002 [2026-10-20 21:56:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1197 [2026-10-20 21:57:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2824 [2026-10-20 21:57:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8355 [2026-10-20 21:57:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5157 [2026-10-20 21:57:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5528 [2026-10-20 21:57:19][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,568 | MFU: 1.8% [2026-10-20 21:57:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4325 [2026-10-20 21:57:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0563 [2026-10-20 21:57:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3435 [2026-10-20 21:57:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5192 [2026-10-20 21:57:24][INFO][training.pretrain_engine] [Ph2|Step 683] loss=5.4852 | ppl=241.10 | aux=0.02509 | lr=3.00e-04 | gnorm=0.691 | tok/s=29,234 | MFU=7.0% | step=143471ms | tokens=0.724B | mem=4.2/32.7GB | cost=$358.95/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 13:36 (135.7h) | Phase3: SKIPPED | SFT: Oct 26 18:21 (140.4h) | DPO (Final): Oct 26 18:55 (141.0h) [2026-10-20 21:57:24][INFO][training.pretrain_engine] Starting global step 684 (accumulating 32 micro-batches)... [2026-10-20 21:57:24][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 21:57:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7122 [2026-10-20 21:57:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8092 [2026-10-20 21:57:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7918 [2026-10-20 21:57:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7539 [2026-10-20 21:57:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3479 [2026-10-20 21:57:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6699 [2026-10-20 21:57:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5247 [2026-10-20 21:57:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3078 [2026-10-20 21:58:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0299 [2026-10-20 21:58:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8154 [2026-10-20 21:58:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5284 [2026-10-20 21:58:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9118 [2026-10-20 21:58:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5909 [2026-10-20 21:58:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1462 [2026-10-20 21:58:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5462 [2026-10-20 21:58:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2551 [2026-10-20 21:58:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6858 [2026-10-20 21:58:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1624 [2026-10-20 21:58:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4214 [2026-10-20 21:58:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3340 [2026-10-20 21:59:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6662 [2026-10-20 21:59:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7604 [2026-10-20 21:59:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5226 [2026-10-20 21:59:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5646 [2026-10-20 21:59:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6806 [2026-10-20 21:59:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5681 [2026-10-20 21:59:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2570 [2026-10-20 21:59:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2961 [2026-10-20 21:59:42][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,568 | MFU: 1.8% [2026-10-20 21:59:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3888 [2026-10-20 21:59:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4544 [2026-10-20 21:59:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1502 [2026-10-20 21:59:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1709 [2026-10-20 21:59:47][INFO][training.pretrain_engine] [Ph2|Step 684] loss=5.5007 | ppl=244.87 | aux=0.02509 | lr=3.00e-04 | gnorm=0.859 | tok/s=29,236 | MFU=7.0% | step=143464ms | tokens=0.725B | mem=4.2/32.7GB | cost=$359.48/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 13:37 (135.6h) | Phase3: SKIPPED | SFT: Oct 26 18:22 (140.4h) | DPO (Final): Oct 26 18:57 (141.0h) [2026-10-20 21:59:47][INFO][training.pretrain_engine] Starting global step 685 (accumulating 32 micro-batches)... [2026-10-20 21:59:47][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 22:00:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1698 [2026-10-20 22:00:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0098 [2026-10-20 22:00:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4952 [2026-10-20 22:00:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3267 [2026-10-20 22:00:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7622 [2026-10-20 22:00:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4634 [2026-10-20 22:00:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2765 [2026-10-20 22:00:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5114 [2026-10-20 22:00:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3863 [2026-10-20 22:00:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5030 [2026-10-20 22:00:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3917 [2026-10-20 22:00:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3205 [2026-10-20 22:00:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8818 [2026-10-20 22:00:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6274 [2026-10-20 22:00:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6654 [2026-10-20 22:00:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5980 [2026-10-20 22:01:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2828 [2026-10-20 22:01:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2530 [2026-10-20 22:01:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7467 [2026-10-20 22:01:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9812 [2026-10-20 22:01:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9157 [2026-10-20 22:01:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4296 [2026-10-20 22:01:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6435 [2026-10-20 22:01:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6310 [2026-10-20 22:01:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5651 [2026-10-20 22:01:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2966 [2026-10-20 22:01:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2143 [2026-10-20 22:01:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3049 [2026-10-20 22:02:05][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,592 | MFU: 1.8% [2026-10-20 22:02:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8251 [2026-10-20 22:02:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7203 [2026-10-20 22:02:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8221 [2026-10-20 22:02:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7138 [2026-10-20 22:02:10][INFO][training.pretrain_engine] [Ph2|Step 685] loss=5.4875 | ppl=241.65 | aux=0.02508 | lr=3.00e-04 | gnorm=0.848 | tok/s=29,237 | MFU=7.0% | step=143459ms | tokens=0.726B | mem=4.2/32.7GB | cost=$360.00/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 13:39 (135.6h) | Phase3: SKIPPED | SFT: Oct 26 18:24 (140.4h) | DPO (Final): Oct 26 18:58 (140.9h) [2026-10-20 22:02:10][INFO][training.pretrain_engine] Starting global step 686 (accumulating 32 micro-batches)... [2026-10-20 22:02:10][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 22:02:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0991 [2026-10-20 22:02:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9173 [2026-10-20 22:02:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1573 [2026-10-20 22:02:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4864 [2026-10-20 22:02:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2840 [2026-10-20 22:02:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2533 [2026-10-20 22:02:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5624 [2026-10-20 22:02:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4753 [2026-10-20 22:03:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8714 [2026-10-20 22:03:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6453 [2026-10-20 22:03:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9326 [2026-10-20 22:03:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4100 [2026-10-20 22:03:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7350 [2026-10-20 22:03:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5087 [2026-10-20 22:03:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8898 [2026-10-20 22:03:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7755 [2026-10-20 22:03:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9818 [2026-10-20 22:03:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4668 [2026-10-20 22:03:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5725 [2026-10-20 22:03:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8158 [2026-10-20 22:03:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3610 [2026-10-20 22:03:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9123 [2026-10-20 22:03:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4215 [2026-10-20 22:03:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3006 [2026-10-20 22:04:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.8215 [2026-10-20 22:04:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1309 [2026-10-20 22:04:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4992 [2026-10-20 22:04:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5992 [2026-10-20 22:04:28][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,593 | MFU: 1.8% [2026-10-20 22:04:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8624 [2026-10-20 22:04:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9253 [2026-10-20 22:04:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7085 [2026-10-20 22:04:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9176 [2026-10-20 22:04:33][INFO][training.pretrain_engine] [Ph2|Step 686] loss=5.4850 | ppl=241.05 | aux=0.02511 | lr=3.00e-04 | gnorm=0.785 | tok/s=29,238 | MFU=7.0% | step=143453ms | tokens=0.727B | mem=4.2/32.7GB | cost=$360.52/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 13:40 (135.6h) | Phase3: SKIPPED | SFT: Oct 26 18:25 (140.4h) | DPO (Final): Oct 26 19:00 (140.9h) [2026-10-20 22:04:36][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step686 | Tokens: 0.727B | Spent: $360.53 | Remaining: $39.47 [2026-10-20 22:04:36][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step686 [2026-10-20 22:04:36][INFO][training.pretrain_engine] Starting global step 687 (accumulating 32 micro-batches)... [2026-10-20 22:04:36][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 22:04:36][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 22:04:36][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 22:04:36][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 22:04:36][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step686 to Phase-Technologies/forge-500m-pretrain (path: phase2_step686)... [2026-10-20 22:04:37][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 22:04:46][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 22:04:46][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step686 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 22:04:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4863 [2026-10-20 22:04:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0444 [2026-10-20 22:04:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8426 [2026-10-20 22:04:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6709 [2026-10-20 22:05:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1787 [2026-10-20 22:05:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2307 [2026-10-20 22:05:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3991 [2026-10-20 22:05:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3828 [2026-10-20 22:05:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4820 [2026-10-20 22:05:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0227 [2026-10-20 22:05:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5037 [2026-10-20 22:05:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5462 [2026-10-20 22:05:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8135 [2026-10-20 22:05:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3655 [2026-10-20 22:05:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9411 [2026-10-20 22:05:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9825 [2026-10-20 22:06:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7931 [2026-10-20 22:06:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4626 [2026-10-20 22:06:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4275 [2026-10-20 22:06:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4635 [2026-10-20 22:06:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4980 [2026-10-20 22:06:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6295 [2026-10-20 22:06:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3444 [2026-10-20 22:06:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6223 [2026-10-20 22:06:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6484 [2026-10-20 22:06:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4650 [2026-10-20 22:06:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5654 [2026-10-20 22:06:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9489 [2026-10-20 22:06:55][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,572 | MFU: 1.8% [2026-10-20 22:06:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0901 [2026-10-20 22:06:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8689 [2026-10-20 22:06:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.6803 [2026-10-20 22:06:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7208 [2026-10-20 22:06:59][INFO][training.pretrain_engine] [Ph2|Step 687] loss=5.5811 | ppl=265.36 | aux=0.02508 | lr=3.00e-04 | gnorm=0.715 | tok/s=29,240 | MFU=7.0% | step=143445ms | tokens=0.728B | mem=4.2/32.7GB | cost=$361.06/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 13:42 (135.6h) | Phase3: SKIPPED | SFT: Oct 26 18:27 (140.3h) | DPO (Final): Oct 26 19:01 (140.9h) [2026-10-20 22:06:59][INFO][training.pretrain_engine] Starting global step 688 (accumulating 32 micro-batches)... [2026-10-20 22:06:59][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 22:07:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3001 [2026-10-20 22:07:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4627 [2026-10-20 22:07:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3308 [2026-10-20 22:07:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3943 [2026-10-20 22:07:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5292 [2026-10-20 22:07:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3827 [2026-10-20 22:07:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8140 [2026-10-20 22:07:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3757 [2026-10-20 22:07:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5046 [2026-10-20 22:07:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6030 [2026-10-20 22:07:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5419 [2026-10-20 22:07:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2888 [2026-10-20 22:08:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9566 [2026-10-20 22:08:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5376 [2026-10-20 22:08:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6974 [2026-10-20 22:08:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2483 [2026-10-20 22:08:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5772 [2026-10-20 22:08:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6666 [2026-10-20 22:08:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6959 [2026-10-20 22:08:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0615 [2026-10-20 22:08:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0896 [2026-10-20 22:08:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6025 [2026-10-20 22:08:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1422 [2026-10-20 22:08:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2024 [2026-10-20 22:09:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0481 [2026-10-20 22:09:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9537 [2026-10-20 22:09:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4168 [2026-10-20 22:09:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8935 [2026-10-20 22:09:17][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,598 | MFU: 1.8% [2026-10-20 22:09:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4499 [2026-10-20 22:09:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6985 [2026-10-20 22:09:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8345 [2026-10-20 22:09:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1392 [2026-10-20 22:09:22][INFO][training.pretrain_engine] [Ph2|Step 688] loss=5.4644 | ppl=236.13 | aux=0.02510 | lr=3.00e-04 | gnorm=0.676 | tok/s=29,241 | MFU=7.0% | step=143441ms | tokens=0.729B | mem=4.2/32.7GB | cost=$361.58/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 13:43 (135.6h) | Phase3: SKIPPED | SFT: Oct 26 18:28 (140.3h) | DPO (Final): Oct 26 19:02 (140.9h) [2026-10-20 22:09:22][INFO][training.pretrain_engine] Starting global step 689 (accumulating 32 micro-batches)... [2026-10-20 22:09:22][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 22:09:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0980 [2026-10-20 22:09:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5204 [2026-10-20 22:09:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8216 [2026-10-20 22:09:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4328 [2026-10-20 22:09:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5385 [2026-10-20 22:09:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4645 [2026-10-20 22:09:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4332 [2026-10-20 22:09:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4630 [2026-10-20 22:10:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7873 [2026-10-20 22:10:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5346 [2026-10-20 22:10:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0314 [2026-10-20 22:10:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7001 [2026-10-20 22:10:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6016 [2026-10-20 22:10:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7169 [2026-10-20 22:10:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6238 [2026-10-20 22:10:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6463 [2026-10-20 22:10:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1312 [2026-10-20 22:10:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1042 [2026-10-20 22:10:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6233 [2026-10-20 22:10:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7243 [2026-10-20 22:11:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8100 [2026-10-20 22:11:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7567 [2026-10-20 22:11:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4340 [2026-10-20 22:11:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7223 [2026-10-20 22:11:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5748 [2026-10-20 22:11:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6516 [2026-10-20 22:11:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2277 [2026-10-20 22:11:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1899 [2026-10-20 22:11:40][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,593 | MFU: 1.8% [2026-10-20 22:11:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4893 [2026-10-20 22:11:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1432 [2026-10-20 22:11:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3270 [2026-10-20 22:11:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3246 [2026-10-20 22:11:46][INFO][training.pretrain_engine] [Ph2|Step 689] loss=5.6411 | ppl=281.76 | aux=0.02508 | lr=3.00e-04 | gnorm=0.578 | tok/s=29,241 | MFU=7.0% | step=143438ms | tokens=0.730B | mem=4.2/32.7GB | cost=$362.11/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 13:45 (135.6h) | Phase3: SKIPPED | SFT: Oct 26 18:30 (140.3h) | DPO (Final): Oct 26 19:04 (140.9h) [2026-10-20 22:11:46][INFO][training.pretrain_engine] Starting global step 690 (accumulating 32 micro-batches)... [2026-10-20 22:11:46][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 22:12:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4897 [2026-10-20 22:12:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8471 [2026-10-20 22:12:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4227 [2026-10-20 22:12:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7654 [2026-10-20 22:12:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2464 [2026-10-20 22:12:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7639 [2026-10-20 22:12:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0765 [2026-10-20 22:12:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5135 [2026-10-20 22:12:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2864 [2026-10-20 22:12:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0090 [2026-10-20 22:12:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6720 [2026-10-20 22:12:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5878 [2026-10-20 22:12:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8383 [2026-10-20 22:12:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6189 [2026-10-20 22:12:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5353 [2026-10-20 22:12:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1390 [2026-10-20 22:13:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4586 [2026-10-20 22:13:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5058 [2026-10-20 22:13:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3994 [2026-10-20 22:13:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9206 [2026-10-20 22:13:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7466 [2026-10-20 22:13:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7890 [2026-10-20 22:13:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0466 [2026-10-20 22:13:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1741 [2026-10-20 22:13:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3976 [2026-10-20 22:13:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4874 [2026-10-20 22:13:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4201 [2026-10-20 22:13:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3595 [2026-10-20 22:14:04][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,585 | MFU: 1.8% [2026-10-20 22:14:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0677 [2026-10-20 22:14:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6252 [2026-10-20 22:14:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8413 [2026-10-20 22:14:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9963 [2026-10-20 22:14:09][INFO][training.pretrain_engine] [Ph2|Step 690] loss=5.5025 | ppl=245.31 | aux=0.02507 | lr=3.00e-04 | gnorm=0.746 | tok/s=29,243 | MFU=7.0% | step=143429ms | tokens=0.731B | mem=4.2/32.7GB | cost=$362.63/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 13:46 (135.5h) | Phase3: SKIPPED | SFT: Oct 26 18:31 (140.3h) | DPO (Final): Oct 26 19:05 (140.9h) [2026-10-20 22:14:09][INFO][training.pretrain_engine] Starting global step 691 (accumulating 32 micro-batches)... [2026-10-20 22:14:09][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 22:14:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5330 [2026-10-20 22:14:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5825 [2026-10-20 22:14:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6612 [2026-10-20 22:14:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7218 [2026-10-20 22:14:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6556 [2026-10-20 22:14:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1438 [2026-10-20 22:14:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3773 [2026-10-20 22:14:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1852 [2026-10-20 22:15:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6945 [2026-10-20 22:15:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5767 [2026-10-20 22:15:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4029 [2026-10-20 22:15:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5077 [2026-10-20 22:15:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1861 [2026-10-20 22:15:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8238 [2026-10-20 22:15:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1835 [2026-10-20 22:15:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8114 [2026-10-20 22:15:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3936 [2026-10-20 22:15:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9123 [2026-10-20 22:15:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9982 [2026-10-20 22:15:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5162 [2026-10-20 22:15:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4914 [2026-10-20 22:15:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4604 [2026-10-20 22:15:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3286 [2026-10-20 22:15:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2494 [2026-10-20 22:16:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9965 [2026-10-20 22:16:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5824 [2026-10-20 22:16:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7750 [2026-10-20 22:16:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6668 [2026-10-20 22:16:27][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,579 | MFU: 1.8% [2026-10-20 22:16:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5623 [2026-10-20 22:16:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4705 [2026-10-20 22:16:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4249 [2026-10-20 22:16:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6373 [2026-10-20 22:16:32][INFO][training.pretrain_engine] [Ph2|Step 691] loss=5.3322 | ppl=206.89 | aux=0.02508 | lr=3.00e-04 | gnorm=0.645 | tok/s=29,244 | MFU=7.0% | step=143424ms | tokens=0.732B | mem=4.2/32.7GB | cost=$363.15/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 13:48 (135.5h) | Phase3: SKIPPED | SFT: Oct 26 18:33 (140.3h) | DPO (Final): Oct 26 19:07 (140.8h) [2026-10-20 22:16:32][INFO][training.pretrain_engine] Starting global step 692 (accumulating 32 micro-batches)... [2026-10-20 22:16:32][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 22:16:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8646 [2026-10-20 22:16:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6364 [2026-10-20 22:16:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5373 [2026-10-20 22:16:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6762 [2026-10-20 22:17:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8490 [2026-10-20 22:17:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8847 [2026-10-20 22:17:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8887 [2026-10-20 22:17:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9657 [2026-10-20 22:17:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4747 [2026-10-20 22:17:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6835 [2026-10-20 22:17:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5992 [2026-10-20 22:17:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5075 [2026-10-20 22:17:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4930 [2026-10-20 22:17:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1692 [2026-10-20 22:17:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0984 [2026-10-20 22:17:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4811 [2026-10-20 22:18:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3804 [2026-10-20 22:18:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9646 [2026-10-20 22:18:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7500 [2026-10-20 22:18:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6693 [2026-10-20 22:18:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8546 [2026-10-20 22:18:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2084 [2026-10-20 22:18:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3154 [2026-10-20 22:18:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1054 [2026-10-20 22:18:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0441 [2026-10-20 22:18:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6090 [2026-10-20 22:18:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5592 [2026-10-20 22:18:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5406 [2026-10-20 22:18:50][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,612 | MFU: 1.8% [2026-10-20 22:18:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4211 [2026-10-20 22:18:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1541 [2026-10-20 22:18:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4398 [2026-10-20 22:18:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6121 [2026-10-20 22:18:55][INFO][training.pretrain_engine] [Ph2|Step 692] loss=5.5416 | ppl=255.09 | aux=0.02507 | lr=3.00e-04 | gnorm=0.613 | tok/s=29,245 | MFU=7.0% | step=143418ms | tokens=0.733B | mem=4.2/32.7GB | cost=$363.67/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 13:49 (135.5h) | Phase3: SKIPPED | SFT: Oct 26 18:34 (140.3h) | DPO (Final): Oct 26 19:08 (140.8h) [2026-10-20 22:18:55][INFO][training.pretrain_engine] Starting global step 693 (accumulating 32 micro-batches)... [2026-10-20 22:18:55][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 22:19:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4826 [2026-10-20 22:19:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5882 [2026-10-20 22:19:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4230 [2026-10-20 22:19:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5557 [2026-10-20 22:19:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0976 [2026-10-20 22:19:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3189 [2026-10-20 22:19:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5860 [2026-10-20 22:19:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4940 [2026-10-20 22:19:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3323 [2026-10-20 22:19:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9037 [2026-10-20 22:19:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6524 [2026-10-20 22:19:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7195 [2026-10-20 22:20:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1895 [2026-10-20 22:20:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2154 [2026-10-20 22:20:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2562 [2026-10-20 22:20:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2414 [2026-10-20 22:20:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1709 [2026-10-20 22:20:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5725 [2026-10-20 22:20:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5722 [2026-10-20 22:20:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8129 [2026-10-20 22:20:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2933 [2026-10-20 22:20:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9380 [2026-10-20 22:20:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5541 [2026-10-20 22:20:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4822 [2026-10-20 22:20:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9854 [2026-10-20 22:20:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6797 [2026-10-20 22:21:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7589 [2026-10-20 22:21:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4021 [2026-10-20 22:21:13][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,576 | MFU: 1.8% [2026-10-20 22:21:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0626 [2026-10-20 22:21:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2763 [2026-10-20 22:21:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1524 [2026-10-20 22:21:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8488 [2026-10-20 22:21:18][INFO][training.pretrain_engine] [Ph2|Step 693] loss=5.5595 | ppl=259.70 | aux=0.02508 | lr=3.00e-04 | gnorm=0.582 | tok/s=29,246 | MFU=7.0% | step=143415ms | tokens=0.734B | mem=4.2/32.7GB | cost=$364.20/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 13:51 (135.5h) | Phase3: SKIPPED | SFT: Oct 26 18:36 (140.2h) | DPO (Final): Oct 26 19:10 (140.8h) [2026-10-20 22:21:18][INFO][training.pretrain_engine] Starting global step 694 (accumulating 32 micro-batches)... [2026-10-20 22:21:18][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 22:21:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1758 [2026-10-20 22:21:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9936 [2026-10-20 22:21:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9714 [2026-10-20 22:21:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7370 [2026-10-20 22:21:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2590 [2026-10-20 22:21:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7732 [2026-10-20 22:21:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.9028 [2026-10-20 22:21:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3507 [2026-10-20 22:22:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8922 [2026-10-20 22:22:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6659 [2026-10-20 22:22:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2473 [2026-10-20 22:22:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9199 [2026-10-20 22:22:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5997 [2026-10-20 22:22:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8030 [2026-10-20 22:22:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7894 [2026-10-20 22:22:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2942 [2026-10-20 22:22:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8100 [2026-10-20 22:22:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6577 [2026-10-20 22:22:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3013 [2026-10-20 22:22:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3439 [2026-10-20 22:23:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2264 [2026-10-20 22:23:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1416 [2026-10-20 22:23:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4188 [2026-10-20 22:23:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8221 [2026-10-20 22:23:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9947 [2026-10-20 22:23:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6587 [2026-10-20 22:23:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1856 [2026-10-20 22:23:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9326 [2026-10-20 22:23:37][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,540 | MFU: 1.8% [2026-10-20 22:23:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3202 [2026-10-20 22:23:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4969 [2026-10-20 22:23:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3329 [2026-10-20 22:23:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2772 [2026-10-20 22:23:41][INFO][training.pretrain_engine] [Ph2|Step 694] loss=5.4497 | ppl=232.70 | aux=0.02508 | lr=3.00e-04 | gnorm=0.539 | tok/s=29,246 | MFU=7.0% | step=143416ms | tokens=0.735B | mem=4.2/32.7GB | cost=$364.72/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 13:53 (135.5h) | Phase3: SKIPPED | SFT: Oct 26 18:38 (140.2h) | DPO (Final): Oct 26 19:12 (140.8h) [2026-10-20 22:23:41][INFO][training.pretrain_engine] Starting global step 695 (accumulating 32 micro-batches)... [2026-10-20 22:23:41][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 22:23:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8363 [2026-10-20 22:23:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2204 [2026-10-20 22:23:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8490 [2026-10-20 22:23:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0767 [2026-10-20 22:24:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6851 [2026-10-20 22:24:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9454 [2026-10-20 22:24:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2074 [2026-10-20 22:24:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4395 [2026-10-20 22:24:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1495 [2026-10-20 22:24:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7356 [2026-10-20 22:24:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7385 [2026-10-20 22:24:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9866 [2026-10-20 22:24:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8852 [2026-10-20 22:24:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5278 [2026-10-20 22:24:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3211 [2026-10-20 22:24:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4151 [2026-10-20 22:25:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4655 [2026-10-20 22:25:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.4206 [2026-10-20 22:25:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7520 [2026-10-20 22:25:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0474 [2026-10-20 22:25:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3877 [2026-10-20 22:25:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3726 [2026-10-20 22:25:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3873 [2026-10-20 22:25:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4984 [2026-10-20 22:25:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5451 [2026-10-20 22:25:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6066 [2026-10-20 22:25:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4174 [2026-10-20 22:25:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5417 [2026-10-20 22:26:00][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,555 | MFU: 1.8% [2026-10-20 22:26:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1762 [2026-10-20 22:26:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5418 [2026-10-20 22:26:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3065 [2026-10-20 22:26:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3493 [2026-10-20 22:26:05][INFO][training.pretrain_engine] [Ph2|Step 695] loss=5.5496 | ppl=257.13 | aux=0.02508 | lr=3.00e-04 | gnorm=0.551 | tok/s=29,247 | MFU=7.0% | step=143412ms | tokens=0.736B | mem=4.2/32.7GB | cost=$365.25/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 13:54 (135.5h) | Phase3: SKIPPED | SFT: Oct 26 18:39 (140.2h) | DPO (Final): Oct 26 19:13 (140.8h) [2026-10-20 22:26:08][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step695 | Tokens: 0.736B | Spent: $365.26 | Remaining: $34.74 [2026-10-20 22:26:08][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step695 [2026-10-20 22:26:08][INFO][training.pretrain_engine] Starting global step 696 (accumulating 32 micro-batches)... [2026-10-20 22:26:08][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 22:26:08][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 22:26:08][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 22:26:08][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 22:26:08][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step695 to Phase-Technologies/forge-500m-pretrain (path: phase2_step695)... [2026-10-20 22:26:09][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 22:26:18][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 22:26:18][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step695 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 22:26:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7426 [2026-10-20 22:26:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5657 [2026-10-20 22:26:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6354 [2026-10-20 22:26:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8323 [2026-10-20 22:26:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9101 [2026-10-20 22:26:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7172 [2026-10-20 22:26:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9962 [2026-10-20 22:26:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0554 [2026-10-20 22:26:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7125 [2026-10-20 22:26:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4263 [2026-10-20 22:26:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5783 [2026-10-20 22:27:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9076 [2026-10-20 22:27:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8043 [2026-10-20 22:27:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4238 [2026-10-20 22:27:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1737 [2026-10-20 22:27:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4057 [2026-10-20 22:27:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4854 [2026-10-20 22:27:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6023 [2026-10-20 22:27:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1849 [2026-10-20 22:27:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6327 [2026-10-20 22:27:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3637 [2026-10-20 22:27:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2267 [2026-10-20 22:27:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2022 [2026-10-20 22:27:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5614 [2026-10-20 22:28:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3018 [2026-10-20 22:28:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4345 [2026-10-20 22:28:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1878 [2026-10-20 22:28:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4689 [2026-10-20 22:28:26][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,578 | MFU: 1.8% [2026-10-20 22:28:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5434 [2026-10-20 22:28:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3785 [2026-10-20 22:28:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2199 [2026-10-20 22:28:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5566 [2026-10-20 22:28:31][INFO][training.pretrain_engine] [Ph2|Step 696] loss=5.4997 | ppl=244.61 | aux=0.02509 | lr=3.00e-04 | gnorm=0.621 | tok/s=29,248 | MFU=7.0% | step=143404ms | tokens=0.737B | mem=4.2/32.7GB | cost=$365.78/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 13:56 (135.5h) | Phase3: SKIPPED | SFT: Oct 26 18:40 (140.2h) | DPO (Final): Oct 26 19:15 (140.8h) [2026-10-20 22:28:31][INFO][training.pretrain_engine] Starting global step 697 (accumulating 32 micro-batches)... [2026-10-20 22:28:31][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 22:28:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6626 [2026-10-20 22:28:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4484 [2026-10-20 22:28:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0213 [2026-10-20 22:28:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5378 [2026-10-20 22:29:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2756 [2026-10-20 22:29:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5665 [2026-10-20 22:29:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4643 [2026-10-20 22:29:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4029 [2026-10-20 22:29:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6243 [2026-10-20 22:29:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7556 [2026-10-20 22:29:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7630 [2026-10-20 22:29:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7099 [2026-10-20 22:29:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1915 [2026-10-20 22:29:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0018 [2026-10-20 22:29:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.1565 [2026-10-20 22:29:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0443 [2026-10-20 22:30:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3566 [2026-10-20 22:30:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7941 [2026-10-20 22:30:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1935 [2026-10-20 22:30:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2690 [2026-10-20 22:30:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6573 [2026-10-20 22:30:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4439 [2026-10-20 22:30:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6391 [2026-10-20 22:30:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0060 [2026-10-20 22:30:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4417 [2026-10-20 22:30:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3923 [2026-10-20 22:30:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3992 [2026-10-20 22:30:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4810 [2026-10-20 22:30:49][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,602 | MFU: 1.8% [2026-10-20 22:30:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5699 [2026-10-20 22:30:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6072 [2026-10-20 22:30:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4888 [2026-10-20 22:30:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1127 [2026-10-20 22:30:54][INFO][training.pretrain_engine] [Ph2|Step 697] loss=5.4454 | ppl=231.70 | aux=0.02508 | lr=3.00e-04 | gnorm=0.652 | tok/s=29,250 | MFU=7.0% | step=143395ms | tokens=0.738B | mem=4.2/32.7GB | cost=$366.30/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 13:57 (135.4h) | Phase3: SKIPPED | SFT: Oct 26 18:42 (140.2h) | DPO (Final): Oct 26 19:16 (140.8h) [2026-10-20 22:30:54][INFO][training.pretrain_engine] Starting global step 698 (accumulating 32 micro-batches)... [2026-10-20 22:30:54][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 22:31:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1297 [2026-10-20 22:31:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4546 [2026-10-20 22:31:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2607 [2026-10-20 22:31:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1888 [2026-10-20 22:31:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3006 [2026-10-20 22:31:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3744 [2026-10-20 22:31:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2150 [2026-10-20 22:31:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9907 [2026-10-20 22:31:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2064 [2026-10-20 22:31:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4686 [2026-10-20 22:31:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3751 [2026-10-20 22:31:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3039 [2026-10-20 22:32:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2342 [2026-10-20 22:32:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2279 [2026-10-20 22:32:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3442 [2026-10-20 22:32:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4538 [2026-10-20 22:32:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6593 [2026-10-20 22:32:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3511 [2026-10-20 22:32:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4113 [2026-10-20 22:32:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3060 [2026-10-20 22:32:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3349 [2026-10-20 22:32:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2740 [2026-10-20 22:32:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6808 [2026-10-20 22:32:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5502 [2026-10-20 22:32:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1480 [2026-10-20 22:32:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7869 [2026-10-20 22:32:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8033 [2026-10-20 22:32:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6862 [2026-10-20 22:33:12][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,585 | MFU: 1.8% [2026-10-20 22:33:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0505 [2026-10-20 22:33:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0062 [2026-10-20 22:33:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8196 [2026-10-20 22:33:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2595 [2026-10-20 22:33:17][INFO][training.pretrain_engine] [Ph2|Step 698] loss=5.4844 | ppl=240.90 | aux=0.02507 | lr=3.00e-04 | gnorm=0.613 | tok/s=29,252 | MFU=7.0% | step=143386ms | tokens=0.739B | mem=4.2/32.7GB | cost=$366.82/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 13:58 (135.4h) | Phase3: SKIPPED | SFT: Oct 26 18:43 (140.2h) | DPO (Final): Oct 26 19:17 (140.7h) [2026-10-20 22:33:17][INFO][training.pretrain_engine] Starting global step 699 (accumulating 32 micro-batches)... [2026-10-20 22:33:17][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 22:33:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3430 [2026-10-20 22:33:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4265 [2026-10-20 22:33:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7017 [2026-10-20 22:33:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6896 [2026-10-20 22:33:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2726 [2026-10-20 22:33:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2435 [2026-10-20 22:33:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7006 [2026-10-20 22:33:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0062 [2026-10-20 22:34:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9193 [2026-10-20 22:34:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1903 [2026-10-20 22:34:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0263 [2026-10-20 22:34:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4665 [2026-10-20 22:34:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7306 [2026-10-20 22:34:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0149 [2026-10-20 22:34:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9291 [2026-10-20 22:34:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9207 [2026-10-20 22:34:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6277 [2026-10-20 22:34:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3171 [2026-10-20 22:34:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6347 [2026-10-20 22:34:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0867 [2026-10-20 22:35:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3754 [2026-10-20 22:35:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6495 [2026-10-20 22:35:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2421 [2026-10-20 22:35:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2268 [2026-10-20 22:35:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2953 [2026-10-20 22:35:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6564 [2026-10-20 22:35:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3002 [2026-10-20 22:35:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0330 [2026-10-20 22:35:35][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,573 | MFU: 1.8% [2026-10-20 22:35:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5436 [2026-10-20 22:35:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2927 [2026-10-20 22:35:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5371 [2026-10-20 22:35:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2554 [2026-10-20 22:35:40][INFO][training.pretrain_engine] [Ph2|Step 699] loss=5.5952 | ppl=269.14 | aux=0.02507 | lr=3.00e-04 | gnorm=0.617 | tok/s=29,253 | MFU=7.0% | step=143380ms | tokens=0.740B | mem=4.2/32.7GB | cost=$367.35/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 14:00 (135.4h) | Phase3: SKIPPED | SFT: Oct 26 18:44 (140.2h) | DPO (Final): Oct 26 19:19 (140.7h) [2026-10-20 22:35:40][INFO][training.pretrain_engine] Starting global step 700 (accumulating 32 micro-batches)... [2026-10-20 22:35:40][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 22:35:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4432 [2026-10-20 22:35:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4017 [2026-10-20 22:35:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6577 [2026-10-20 22:35:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3140 [2026-10-20 22:36:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8635 [2026-10-20 22:36:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1244 [2026-10-20 22:36:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0122 [2026-10-20 22:36:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0422 [2026-10-20 22:36:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2920 [2026-10-20 22:36:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0067 [2026-10-20 22:36:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7910 [2026-10-20 22:36:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5139 [2026-10-20 22:36:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3153 [2026-10-20 22:36:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8881 [2026-10-20 22:36:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3709 [2026-10-20 22:36:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7679 [2026-10-20 22:37:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0082 [2026-10-20 22:37:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8453 [2026-10-20 22:37:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7593 [2026-10-20 22:37:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5255 [2026-10-20 22:37:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5794 [2026-10-20 22:37:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6804 [2026-10-20 22:37:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4572 [2026-10-20 22:37:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1417 [2026-10-20 22:37:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5544 [2026-10-20 22:37:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4157 [2026-10-20 22:37:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4618 [2026-10-20 22:37:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2315 [2026-10-20 22:37:58][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,565 | MFU: 1.8% [2026-10-20 22:38:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2907 [2026-10-20 22:38:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0222 [2026-10-20 22:38:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5085 [2026-10-20 22:38:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3880 [2026-10-20 22:38:03][INFO][training.pretrain_engine] [Ph2|Step 700] loss=5.5367 | ppl=253.84 | aux=0.02507 | lr=3.00e-04 | gnorm=0.668 | tok/s=29,254 | MFU=7.0% | step=143376ms | tokens=0.741B | mem=4.2/32.7GB | cost=$367.87/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 14:01 (135.4h) | Phase3: SKIPPED | SFT: Oct 26 18:46 (140.1h) | DPO (Final): Oct 26 19:20 (140.7h) [2026-10-20 22:38:03][INFO][training.pretrain_engine] Starting global step 701 (accumulating 32 micro-batches)... [2026-10-20 22:38:03][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 22:38:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7784 [2026-10-20 22:38:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6216 [2026-10-20 22:38:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6981 [2026-10-20 22:38:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1892 [2026-10-20 22:38:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3367 [2026-10-20 22:38:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3752 [2026-10-20 22:38:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1901 [2026-10-20 22:38:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5631 [2026-10-20 22:38:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0262 [2026-10-20 22:38:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9034 [2026-10-20 22:38:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5459 [2026-10-20 22:38:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3605 [2026-10-20 22:39:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6798 [2026-10-20 22:39:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7182 [2026-10-20 22:39:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7292 [2026-10-20 22:39:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7302 [2026-10-20 22:39:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8311 [2026-10-20 22:39:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5280 [2026-10-20 22:39:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8258 [2026-10-20 22:39:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0327 [2026-10-20 22:39:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5769 [2026-10-20 22:39:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1170 [2026-10-20 22:39:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4677 [2026-10-20 22:39:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3312 [2026-10-20 22:40:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4276 [2026-10-20 22:40:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7061 [2026-10-20 22:40:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3773 [2026-10-20 22:40:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4138 [2026-10-20 22:40:22][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,552 | MFU: 1.8% [2026-10-20 22:40:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4525 [2026-10-20 22:40:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2171 [2026-10-20 22:40:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8504 [2026-10-20 22:40:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2500 [2026-10-20 22:40:26][INFO][training.pretrain_engine] [Ph2|Step 701] loss=5.5596 | ppl=259.71 | aux=0.02509 | lr=3.00e-04 | gnorm=0.855 | tok/s=29,255 | MFU=7.0% | step=143370ms | tokens=0.742B | mem=4.2/32.7GB | cost=$368.39/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 14:03 (135.4h) | Phase3: SKIPPED | SFT: Oct 26 18:47 (140.1h) | DPO (Final): Oct 26 19:22 (140.7h) [2026-10-20 22:40:26][INFO][training.pretrain_engine] Starting global step 702 (accumulating 32 micro-batches)... [2026-10-20 22:40:26][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 22:40:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5945 [2026-10-20 22:40:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1718 [2026-10-20 22:40:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4644 [2026-10-20 22:40:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8155 [2026-10-20 22:41:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3369 [2026-10-20 22:41:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5215 [2026-10-20 22:41:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5826 [2026-10-20 22:41:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5635 [2026-10-20 22:41:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7782 [2026-10-20 22:41:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7461 [2026-10-20 22:41:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4228 [2026-10-20 22:41:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5714 [2026-10-20 22:41:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1512 [2026-10-20 22:41:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4524 [2026-10-20 22:41:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3311 [2026-10-20 22:41:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0366 [2026-10-20 22:41:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2400 [2026-10-20 22:41:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2104 [2026-10-20 22:41:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7764 [2026-10-20 22:41:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1123 [2026-10-20 22:42:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2001 [2026-10-20 22:42:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6187 [2026-10-20 22:42:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3541 [2026-10-20 22:42:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1656 [2026-10-20 22:42:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6979 [2026-10-20 22:42:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8144 [2026-10-20 22:42:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7287 [2026-10-20 22:42:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6479 [2026-10-20 22:42:45][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,573 | MFU: 1.8% [2026-10-20 22:42:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8797 [2026-10-20 22:42:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9416 [2026-10-20 22:42:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5547 [2026-10-20 22:42:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3688 [2026-10-20 22:42:49][INFO][training.pretrain_engine] [Ph2|Step 702] loss=5.4887 | ppl=241.95 | aux=0.02506 | lr=3.00e-04 | gnorm=1.359 | tok/s=29,257 | MFU=7.0% | step=143359ms | tokens=0.743B | mem=4.2/32.7GB | cost=$368.92/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 14:04 (135.4h) | Phase3: SKIPPED | SFT: Oct 26 18:48 (140.1h) | DPO (Final): Oct 26 19:23 (140.7h) [2026-10-20 22:42:49][INFO][training.pretrain_engine] Starting global step 703 (accumulating 32 micro-batches)... [2026-10-20 22:42:49][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 22:43:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6292 [2026-10-20 22:43:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1412 [2026-10-20 22:43:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7456 [2026-10-20 22:43:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9736 [2026-10-20 22:43:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5827 [2026-10-20 22:43:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7844 [2026-10-20 22:43:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5863 [2026-10-20 22:43:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4075 [2026-10-20 22:43:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3599 [2026-10-20 22:43:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0252 [2026-10-20 22:43:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5151 [2026-10-20 22:43:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9976 [2026-10-20 22:44:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6330 [2026-10-20 22:44:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4705 [2026-10-20 22:44:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2733 [2026-10-20 22:44:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6053 [2026-10-20 22:44:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3257 [2026-10-20 22:44:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3956 [2026-10-20 22:44:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6811 [2026-10-20 22:44:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7459 [2026-10-20 22:44:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6312 [2026-10-20 22:44:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5701 [2026-10-20 22:44:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6283 [2026-10-20 22:44:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8430 [2026-10-20 22:44:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1300 [2026-10-20 22:44:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5744 [2026-10-20 22:44:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7535 [2026-10-20 22:44:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8288 [2026-10-20 22:45:08][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,542 | MFU: 1.8% [2026-10-20 22:45:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1872 [2026-10-20 22:45:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2750 [2026-10-20 22:45:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9180 [2026-10-20 22:45:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0002 [2026-10-20 22:45:13][INFO][training.pretrain_engine] [Ph2|Step 703] loss=5.4408 | ppl=230.62 | aux=0.02508 | lr=3.00e-04 | gnorm=0.848 | tok/s=29,257 | MFU=7.0% | step=143361ms | tokens=0.744B | mem=4.2/32.7GB | cost=$369.44/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 14:06 (135.3h) | Phase3: SKIPPED | SFT: Oct 26 18:50 (140.1h) | DPO (Final): Oct 26 19:25 (140.7h) [2026-10-20 22:45:13][INFO][training.pretrain_engine] Starting global step 704 (accumulating 32 micro-batches)... [2026-10-20 22:45:13][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 22:45:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2323 [2026-10-20 22:45:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0928 [2026-10-20 22:45:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9465 [2026-10-20 22:45:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5221 [2026-10-20 22:45:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3173 [2026-10-20 22:45:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1733 [2026-10-20 22:45:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6774 [2026-10-20 22:45:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6244 [2026-10-20 22:46:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5728 [2026-10-20 22:46:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4163 [2026-10-20 22:46:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3708 [2026-10-20 22:46:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4779 [2026-10-20 22:46:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2596 [2026-10-20 22:46:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3734 [2026-10-20 22:46:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4587 [2026-10-20 22:46:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4521 [2026-10-20 22:46:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4249 [2026-10-20 22:46:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2450 [2026-10-20 22:46:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4350 [2026-10-20 22:46:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4249 [2026-10-20 22:47:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2926 [2026-10-20 22:47:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9996 [2026-10-20 22:47:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3155 [2026-10-20 22:47:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3846 [2026-10-20 22:47:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2681 [2026-10-20 22:47:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4630 [2026-10-20 22:47:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4836 [2026-10-20 22:47:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3107 [2026-10-20 22:47:32][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,539 | MFU: 1.8% [2026-10-20 22:47:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2820 [2026-10-20 22:47:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6073 [2026-10-20 22:47:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3860 [2026-10-20 22:47:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2856 [2026-10-20 22:47:36][INFO][training.pretrain_engine] [Ph2|Step 704] loss=5.5439 | ppl=255.68 | aux=0.02507 | lr=3.00e-04 | gnorm=0.785 | tok/s=29,258 | MFU=7.0% | step=143355ms | tokens=0.746B | mem=4.2/32.7GB | cost=$369.97/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 14:07 (135.3h) | Phase3: SKIPPED | SFT: Oct 26 18:52 (140.1h) | DPO (Final): Oct 26 19:26 (140.6h) [2026-10-20 22:47:40][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step704 | Tokens: 0.746B | Spent: $369.98 | Remaining: $30.02 [2026-10-20 22:47:40][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step704 [2026-10-20 22:47:40][INFO][training.pretrain_engine] Starting global step 705 (accumulating 32 micro-batches)... [2026-10-20 22:47:40][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 22:47:40][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 22:47:40][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 22:47:40][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 22:47:40][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step704 to Phase-Technologies/forge-500m-pretrain (path: phase2_step704)... [2026-10-20 22:47:40][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 22:47:49][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 22:47:49][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step704 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 22:47:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4245 [2026-10-20 22:47:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7885 [2026-10-20 22:47:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8346 [2026-10-20 22:47:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5875 [2026-10-20 22:48:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5730 [2026-10-20 22:48:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6374 [2026-10-20 22:48:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6749 [2026-10-20 22:48:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8950 [2026-10-20 22:48:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9187 [2026-10-20 22:48:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7744 [2026-10-20 22:48:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2444 [2026-10-20 22:48:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0852 [2026-10-20 22:48:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2951 [2026-10-20 22:48:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5983 [2026-10-20 22:48:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5756 [2026-10-20 22:48:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5295 [2026-10-20 22:49:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9414 [2026-10-20 22:49:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6778 [2026-10-20 22:49:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5450 [2026-10-20 22:49:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8480 [2026-10-20 22:49:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3791 [2026-10-20 22:49:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2691 [2026-10-20 22:49:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3191 [2026-10-20 22:49:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6074 [2026-10-20 22:49:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6375 [2026-10-20 22:49:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0910 [2026-10-20 22:49:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7736 [2026-10-20 22:49:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7999 [2026-10-20 22:49:59][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,549 | MFU: 1.8% [2026-10-20 22:50:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5750 [2026-10-20 22:50:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6308 [2026-10-20 22:50:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6028 [2026-10-20 22:50:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6301 [2026-10-20 22:50:03][INFO][training.pretrain_engine] [Ph2|Step 705] loss=5.5331 | ppl=252.93 | aux=0.02509 | lr=3.00e-04 | gnorm=1.000 | tok/s=29,259 | MFU=7.0% | step=143350ms | tokens=0.747B | mem=4.2/32.7GB | cost=$370.50/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 14:09 (135.3h) | Phase3: SKIPPED | SFT: Oct 26 18:53 (140.1h) | DPO (Final): Oct 26 19:28 (140.6h) [2026-10-20 22:50:03][INFO][training.pretrain_engine] Starting global step 706 (accumulating 32 micro-batches)... [2026-10-20 22:50:03][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 22:50:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4613 [2026-10-20 22:50:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9385 [2026-10-20 22:50:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4395 [2026-10-20 22:50:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1498 [2026-10-20 22:50:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7030 [2026-10-20 22:50:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6689 [2026-10-20 22:50:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5845 [2026-10-20 22:50:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6804 [2026-10-20 22:50:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2651 [2026-10-20 22:50:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4886 [2026-10-20 22:50:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5826 [2026-10-20 22:50:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2602 [2026-10-20 22:51:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3019 [2026-10-20 22:51:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8701 [2026-10-20 22:51:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1617 [2026-10-20 22:51:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2572 [2026-10-20 22:51:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0842 [2026-10-20 22:51:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0186 [2026-10-20 22:51:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8786 [2026-10-20 22:51:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1695 [2026-10-20 22:51:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5854 [2026-10-20 22:51:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1540 [2026-10-20 22:51:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5169 [2026-10-20 22:51:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6345 [2026-10-20 22:52:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1077 [2026-10-20 22:52:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9630 [2026-10-20 22:52:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7149 [2026-10-20 22:52:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3729 [2026-10-20 22:52:22][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,568 | MFU: 1.8% [2026-10-20 22:52:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5370 [2026-10-20 22:52:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3227 [2026-10-20 22:52:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7081 [2026-10-20 22:52:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5381 [2026-10-20 22:52:26][INFO][training.pretrain_engine] [Ph2|Step 706] loss=5.5793 | ppl=264.89 | aux=0.02508 | lr=3.00e-04 | gnorm=1.219 | tok/s=29,261 | MFU=7.0% | step=143342ms | tokens=0.748B | mem=4.2/32.7GB | cost=$371.03/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 14:10 (135.3h) | Phase3: SKIPPED | SFT: Oct 26 18:55 (140.0h) | DPO (Final): Oct 26 19:29 (140.6h) [2026-10-20 22:52:26][INFO][training.pretrain_engine] Starting global step 707 (accumulating 32 micro-batches)... [2026-10-20 22:52:26][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 22:52:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5700 [2026-10-20 22:52:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5748 [2026-10-20 22:52:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3293 [2026-10-20 22:52:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3806 [2026-10-20 22:53:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5287 [2026-10-20 22:53:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0458 [2026-10-20 22:53:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.5033 [2026-10-20 22:53:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4639 [2026-10-20 22:53:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2357 [2026-10-20 22:53:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1815 [2026-10-20 22:53:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3877 [2026-10-20 22:53:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4089 [2026-10-20 22:53:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0669 [2026-10-20 22:53:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9200 [2026-10-20 22:53:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7216 [2026-10-20 22:53:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2228 [2026-10-20 22:53:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5203 [2026-10-20 22:53:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6792 [2026-10-20 22:53:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1564 [2026-10-20 22:53:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4052 [2026-10-20 22:54:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2133 [2026-10-20 22:54:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2923 [2026-10-20 22:54:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0212 [2026-10-20 22:54:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3064 [2026-10-20 22:54:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9704 [2026-10-20 22:54:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3358 [2026-10-20 22:54:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1117 [2026-10-20 22:54:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3998 [2026-10-20 22:54:45][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,563 | MFU: 1.8% [2026-10-20 22:54:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2739 [2026-10-20 22:54:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5435 [2026-10-20 22:54:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6262 [2026-10-20 22:54:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6044 [2026-10-20 22:54:49][INFO][training.pretrain_engine] [Ph2|Step 707] loss=5.5440 | ppl=255.69 | aux=0.02509 | lr=3.00e-04 | gnorm=1.086 | tok/s=29,262 | MFU=7.0% | step=143337ms | tokens=0.749B | mem=4.2/32.7GB | cost=$371.55/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 14:11 (135.3h) | Phase3: SKIPPED | SFT: Oct 26 18:56 (140.0h) | DPO (Final): Oct 26 19:30 (140.6h) [2026-10-20 22:54:49][INFO][training.pretrain_engine] Starting global step 708 (accumulating 32 micro-batches)... [2026-10-20 22:54:49][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 22:55:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7212 [2026-10-20 22:55:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5285 [2026-10-20 22:55:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0545 [2026-10-20 22:55:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8321 [2026-10-20 22:55:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9278 [2026-10-20 22:55:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6015 [2026-10-20 22:55:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7751 [2026-10-20 22:55:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8882 [2026-10-20 22:55:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7767 [2026-10-20 22:55:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1117 [2026-10-20 22:55:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2990 [2026-10-20 22:55:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2209 [2026-10-20 22:56:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3810 [2026-10-20 22:56:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6719 [2026-10-20 22:56:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9977 [2026-10-20 22:56:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7971 [2026-10-20 22:56:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3498 [2026-10-20 22:56:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5055 [2026-10-20 22:56:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5338 [2026-10-20 22:56:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7969 [2026-10-20 22:56:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0871 [2026-10-20 22:56:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0505 [2026-10-20 22:56:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0169 [2026-10-20 22:56:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6507 [2026-10-20 22:56:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9150 [2026-10-20 22:56:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7791 [2026-10-20 22:56:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8762 [2026-10-20 22:56:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7179 [2026-10-20 22:57:08][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,581 | MFU: 1.8% [2026-10-20 22:57:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4525 [2026-10-20 22:57:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7299 [2026-10-20 22:57:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7076 [2026-10-20 22:57:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4470 [2026-10-20 22:57:13][INFO][training.pretrain_engine] [Ph2|Step 708] loss=5.4838 | ppl=240.76 | aux=0.02509 | lr=3.00e-04 | gnorm=0.902 | tok/s=29,264 | MFU=7.0% | step=143328ms | tokens=0.750B | mem=4.2/32.7GB | cost=$372.07/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 14:13 (135.3h) | Phase3: SKIPPED | SFT: Oct 26 18:58 (140.0h) | DPO (Final): Oct 26 19:32 (140.6h) [2026-10-20 22:57:13][INFO][training.pretrain_engine] Starting global step 709 (accumulating 32 micro-batches)... [2026-10-20 22:57:13][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 22:57:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1877 [2026-10-20 22:57:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2770 [2026-10-20 22:57:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0501 [2026-10-20 22:57:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9701 [2026-10-20 22:57:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3517 [2026-10-20 22:57:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7035 [2026-10-20 22:57:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4771 [2026-10-20 22:57:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6183 [2026-10-20 22:58:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7908 [2026-10-20 22:58:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7460 [2026-10-20 22:58:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1220 [2026-10-20 22:58:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4667 [2026-10-20 22:58:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4907 [2026-10-20 22:58:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7293 [2026-10-20 22:58:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8380 [2026-10-20 22:58:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0643 [2026-10-20 22:58:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7521 [2026-10-20 22:58:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5302 [2026-10-20 22:58:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2591 [2026-10-20 22:58:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5514 [2026-10-20 22:58:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1117 [2026-10-20 22:58:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1591 [2026-10-20 22:58:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3784 [2026-10-20 22:59:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7332 [2026-10-20 22:59:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8748 [2026-10-20 22:59:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5762 [2026-10-20 22:59:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2494 [2026-10-20 22:59:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5262 [2026-10-20 22:59:31][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,579 | MFU: 1.8% [2026-10-20 22:59:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3888 [2026-10-20 22:59:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3627 [2026-10-20 22:59:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3557 [2026-10-20 22:59:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4942 [2026-10-20 22:59:35][INFO][training.pretrain_engine] [Ph2|Step 709] loss=5.4561 | ppl=234.19 | aux=0.02508 | lr=3.00e-04 | gnorm=0.676 | tok/s=29,265 | MFU=7.0% | step=143320ms | tokens=0.751B | mem=4.2/32.7GB | cost=$372.60/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 14:14 (135.2h) | Phase3: SKIPPED | SFT: Oct 26 18:59 (140.0h) | DPO (Final): Oct 26 19:33 (140.6h) [2026-10-20 22:59:35][INFO][training.pretrain_engine] Starting global step 710 (accumulating 32 micro-batches)... [2026-10-20 22:59:35][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 22:59:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7735 [2026-10-20 22:59:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4912 [2026-10-20 22:59:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5153 [2026-10-20 22:59:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5730 [2026-10-20 23:00:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2828 [2026-10-20 23:00:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7845 [2026-10-20 23:00:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4738 [2026-10-20 23:00:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1093 [2026-10-20 23:00:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2576 [2026-10-20 23:00:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1225 [2026-10-20 23:00:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9834 [2026-10-20 23:00:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3258 [2026-10-20 23:00:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4520 [2026-10-20 23:00:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2827 [2026-10-20 23:00:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6858 [2026-10-20 23:00:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8764 [2026-10-20 23:01:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5212 [2026-10-20 23:01:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5315 [2026-10-20 23:01:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5164 [2026-10-20 23:01:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5151 [2026-10-20 23:01:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5737 [2026-10-20 23:01:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5078 [2026-10-20 23:01:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5337 [2026-10-20 23:01:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8862 [2026-10-20 23:01:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1318 [2026-10-20 23:01:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5315 [2026-10-20 23:01:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0335 [2026-10-20 23:01:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4622 [2026-10-20 23:01:54][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,595 | MFU: 1.8% [2026-10-20 23:01:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1826 [2026-10-20 23:01:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3233 [2026-10-20 23:01:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8742 [2026-10-20 23:01:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0006 [2026-10-20 23:01:59][INFO][training.pretrain_engine] [Ph2|Step 710] loss=5.4578 | ppl=234.57 | aux=0.02508 | lr=3.00e-04 | gnorm=0.922 | tok/s=29,266 | MFU=7.0% | step=143314ms | tokens=0.752B | mem=4.2/32.7GB | cost=$373.12/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 14:16 (135.2h) | Phase3: SKIPPED | SFT: Oct 26 19:00 (140.0h) | DPO (Final): Oct 26 19:34 (140.5h) [2026-10-20 23:01:59][INFO][training.pretrain_engine] Starting global step 711 (accumulating 32 micro-batches)... [2026-10-20 23:01:59][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 23:02:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1399 [2026-10-20 23:02:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3477 [2026-10-20 23:02:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5522 [2026-10-20 23:02:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9419 [2026-10-20 23:02:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5895 [2026-10-20 23:02:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3653 [2026-10-20 23:02:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5298 [2026-10-20 23:02:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5163 [2026-10-20 23:02:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0674 [2026-10-20 23:02:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2677 [2026-10-20 23:02:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2581 [2026-10-20 23:02:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0783 [2026-10-20 23:03:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5322 [2026-10-20 23:03:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6682 [2026-10-20 23:03:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7388 [2026-10-20 23:03:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7019 [2026-10-20 23:03:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7442 [2026-10-20 23:03:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8048 [2026-10-20 23:03:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3194 [2026-10-20 23:03:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4240 [2026-10-20 23:03:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3223 [2026-10-20 23:03:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6555 [2026-10-20 23:03:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1638 [2026-10-20 23:03:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7767 [2026-10-20 23:04:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2883 [2026-10-20 23:04:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3361 [2026-10-20 23:04:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8349 [2026-10-20 23:04:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7223 [2026-10-20 23:04:17][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,586 | MFU: 1.8% [2026-10-20 23:04:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4260 [2026-10-20 23:04:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6253 [2026-10-20 23:04:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6801 [2026-10-20 23:04:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7901 [2026-10-20 23:04:21][INFO][training.pretrain_engine] [Ph2|Step 711] loss=5.4806 | ppl=239.99 | aux=0.02507 | lr=3.00e-04 | gnorm=0.574 | tok/s=29,269 | MFU=7.0% | step=143304ms | tokens=0.753B | mem=4.2/32.7GB | cost=$373.64/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 14:17 (135.2h) | Phase3: SKIPPED | SFT: Oct 26 19:01 (140.0h) | DPO (Final): Oct 26 19:36 (140.5h) [2026-10-20 23:04:21][INFO][training.pretrain_engine] Starting global step 712 (accumulating 32 micro-batches)... [2026-10-20 23:04:21][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 23:04:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0769 [2026-10-20 23:04:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7816 [2026-10-20 23:04:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1940 [2026-10-20 23:04:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3708 [2026-10-20 23:04:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4930 [2026-10-20 23:04:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1013 [2026-10-20 23:04:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2604 [2026-10-20 23:04:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9397 [2026-10-20 23:05:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1364 [2026-10-20 23:05:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5316 [2026-10-20 23:05:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8500 [2026-10-20 23:05:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7180 [2026-10-20 23:05:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2936 [2026-10-20 23:05:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0790 [2026-10-20 23:05:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2761 [2026-10-20 23:05:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6553 [2026-10-20 23:05:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9785 [2026-10-20 23:05:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9917 [2026-10-20 23:05:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9346 [2026-10-20 23:05:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8287 [2026-10-20 23:06:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7389 [2026-10-20 23:06:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6123 [2026-10-20 23:06:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1599 [2026-10-20 23:06:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5286 [2026-10-20 23:06:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1370 [2026-10-20 23:06:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1739 [2026-10-20 23:06:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3888 [2026-10-20 23:06:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3262 [2026-10-20 23:06:39][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,622 | MFU: 1.8% [2026-10-20 23:06:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5027 [2026-10-20 23:06:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0449 [2026-10-20 23:06:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8837 [2026-10-20 23:06:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9700 [2026-10-20 23:06:44][INFO][training.pretrain_engine] [Ph2|Step 712] loss=5.5650 | ppl=261.13 | aux=0.02508 | lr=3.00e-04 | gnorm=0.773 | tok/s=29,270 | MFU=7.0% | step=143295ms | tokens=0.754B | mem=4.2/32.7GB | cost=$374.16/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 14:18 (135.2h) | Phase3: SKIPPED | SFT: Oct 26 19:03 (139.9h) | DPO (Final): Oct 26 19:37 (140.5h) [2026-10-20 23:06:44][INFO][training.pretrain_engine] Starting global step 713 (accumulating 32 micro-batches)... [2026-10-20 23:06:44][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 23:07:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1853 [2026-10-20 23:07:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1717 [2026-10-20 23:07:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0925 [2026-10-20 23:07:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9421 [2026-10-20 23:07:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3883 [2026-10-20 23:07:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9511 [2026-10-20 23:07:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0229 [2026-10-20 23:07:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7345 [2026-10-20 23:07:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8998 [2026-10-20 23:07:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7211 [2026-10-20 23:07:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3457 [2026-10-20 23:07:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3241 [2026-10-20 23:07:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7530 [2026-10-20 23:07:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3365 [2026-10-20 23:07:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8927 [2026-10-20 23:07:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4834 [2026-10-20 23:08:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4778 [2026-10-20 23:08:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3155 [2026-10-20 23:08:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1830 [2026-10-20 23:08:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5209 [2026-10-20 23:08:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1507 [2026-10-20 23:08:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3369 [2026-10-20 23:08:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9430 [2026-10-20 23:08:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5604 [2026-10-20 23:08:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5560 [2026-10-20 23:08:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1498 [2026-10-20 23:08:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3812 [2026-10-20 23:08:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3038 [2026-10-20 23:09:02][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,599 | MFU: 1.8% [2026-10-20 23:09:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7188 [2026-10-20 23:09:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8759 [2026-10-20 23:09:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9193 [2026-10-20 23:09:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6230 [2026-10-20 23:09:07][INFO][training.pretrain_engine] [Ph2|Step 713] loss=5.4792 | ppl=239.66 | aux=0.02508 | lr=3.00e-04 | gnorm=0.746 | tok/s=29,272 | MFU=7.0% | step=143288ms | tokens=0.755B | mem=4.2/32.7GB | cost=$374.68/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 14:19 (135.2h) | Phase3: SKIPPED | SFT: Oct 26 19:04 (139.9h) | DPO (Final): Oct 26 19:38 (140.5h) [2026-10-20 23:09:10][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step713 | Tokens: 0.755B | Spent: $374.70 | Remaining: $25.30 [2026-10-20 23:09:10][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step713 [2026-10-20 23:09:10][INFO][training.pretrain_engine] Starting global step 714 (accumulating 32 micro-batches)... [2026-10-20 23:09:10][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 23:09:10][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 23:09:10][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 23:09:10][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 23:09:10][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step713 to Phase-Technologies/forge-500m-pretrain (path: phase2_step713)... [2026-10-20 23:09:10][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 23:09:19][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 23:09:19][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step713 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 23:09:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5843 [2026-10-20 23:09:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6191 [2026-10-20 23:09:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5077 [2026-10-20 23:09:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5980 [2026-10-20 23:09:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3851 [2026-10-20 23:09:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0861 [2026-10-20 23:09:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5155 [2026-10-20 23:09:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2266 [2026-10-20 23:10:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1627 [2026-10-20 23:10:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4153 [2026-10-20 23:10:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1985 [2026-10-20 23:10:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0037 [2026-10-20 23:10:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2675 [2026-10-20 23:10:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4318 [2026-10-20 23:10:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5949 [2026-10-20 23:10:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7400 [2026-10-20 23:10:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3660 [2026-10-20 23:10:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1245 [2026-10-20 23:10:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5476 [2026-10-20 23:10:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5167 [2026-10-20 23:10:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7608 [2026-10-20 23:10:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5662 [2026-10-20 23:10:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1604 [2026-10-20 23:10:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5927 [2026-10-20 23:11:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1104 [2026-10-20 23:11:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1264 [2026-10-20 23:11:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2531 [2026-10-20 23:11:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0845 [2026-10-20 23:11:28][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,619 | MFU: 1.8% [2026-10-20 23:11:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8781 [2026-10-20 23:11:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8795 [2026-10-20 23:11:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9486 [2026-10-20 23:11:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2490 [2026-10-20 23:11:32][INFO][training.pretrain_engine] [Ph2|Step 714] loss=5.4925 | ppl=242.85 | aux=0.02508 | lr=3.00e-04 | gnorm=0.789 | tok/s=29,275 | MFU=7.0% | step=143273ms | tokens=0.756B | mem=4.2/32.7GB | cost=$375.22/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 14:20 (135.2h) | Phase3: SKIPPED | SFT: Oct 26 19:05 (139.9h) | DPO (Final): Oct 26 19:39 (140.5h) [2026-10-20 23:11:32][INFO][training.pretrain_engine] Starting global step 715 (accumulating 32 micro-batches)... [2026-10-20 23:11:32][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 23:11:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6976 [2026-10-20 23:11:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7496 [2026-10-20 23:11:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7856 [2026-10-20 23:11:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4290 [2026-10-20 23:12:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1352 [2026-10-20 23:12:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8568 [2026-10-20 23:12:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2637 [2026-10-20 23:12:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0882 [2026-10-20 23:12:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8817 [2026-10-20 23:12:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7927 [2026-10-20 23:12:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5307 [2026-10-20 23:12:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5039 [2026-10-20 23:12:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3736 [2026-10-20 23:12:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0323 [2026-10-20 23:12:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7963 [2026-10-20 23:12:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4306 [2026-10-20 23:13:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1964 [2026-10-20 23:13:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4705 [2026-10-20 23:13:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2023 [2026-10-20 23:13:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4257 [2026-10-20 23:13:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1094 [2026-10-20 23:13:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3320 [2026-10-20 23:13:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9237 [2026-10-20 23:13:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3822 [2026-10-20 23:13:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6389 [2026-10-20 23:13:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5389 [2026-10-20 23:13:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4747 [2026-10-20 23:13:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6804 [2026-10-20 23:13:50][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,583 | MFU: 1.8% [2026-10-20 23:13:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2126 [2026-10-20 23:13:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3760 [2026-10-20 23:13:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2298 [2026-10-20 23:13:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3833 [2026-10-20 23:13:55][INFO][training.pretrain_engine] [Ph2|Step 715] loss=5.3661 | ppl=214.03 | aux=0.02508 | lr=3.00e-04 | gnorm=0.754 | tok/s=29,277 | MFU=7.0% | step=143260ms | tokens=0.757B | mem=4.2/32.7GB | cost=$375.74/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 14:21 (135.1h) | Phase3: SKIPPED | SFT: Oct 26 19:06 (139.9h) | DPO (Final): Oct 26 19:40 (140.4h) [2026-10-20 23:13:55][INFO][training.pretrain_engine] Starting global step 716 (accumulating 32 micro-batches)... [2026-10-20 23:13:55][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 23:14:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1784 [2026-10-20 23:14:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5384 [2026-10-20 23:14:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.4504 [2026-10-20 23:14:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8865 [2026-10-20 23:14:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0045 [2026-10-20 23:14:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4032 [2026-10-20 23:14:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9035 [2026-10-20 23:14:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7977 [2026-10-20 23:14:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3166 [2026-10-20 23:14:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4639 [2026-10-20 23:14:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4921 [2026-10-20 23:14:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3537 [2026-10-20 23:15:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3955 [2026-10-20 23:15:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0984 [2026-10-20 23:15:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6669 [2026-10-20 23:15:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5938 [2026-10-20 23:15:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8211 [2026-10-20 23:15:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7055 [2026-10-20 23:15:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6208 [2026-10-20 23:15:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7836 [2026-10-20 23:15:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1513 [2026-10-20 23:15:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6250 [2026-10-20 23:15:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4870 [2026-10-20 23:15:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6829 [2026-10-20 23:16:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5741 [2026-10-20 23:16:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2415 [2026-10-20 23:16:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1124 [2026-10-20 23:16:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3990 [2026-10-20 23:16:14][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,579 | MFU: 1.8% [2026-10-20 23:16:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7270 [2026-10-20 23:16:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9701 [2026-10-20 23:16:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1340 [2026-10-20 23:16:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7449 [2026-10-20 23:16:18][INFO][training.pretrain_engine] [Ph2|Step 716] loss=5.4144 | ppl=224.61 | aux=0.02508 | lr=3.00e-04 | gnorm=0.891 | tok/s=29,280 | MFU=7.0% | step=143249ms | tokens=0.758B | mem=4.2/32.7GB | cost=$376.26/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 14:23 (135.1h) | Phase3: SKIPPED | SFT: Oct 26 19:07 (139.9h) | DPO (Final): Oct 26 19:41 (140.4h) [2026-10-20 23:16:18][INFO][training.pretrain_engine] Starting global step 717 (accumulating 32 micro-batches)... [2026-10-20 23:16:18][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 23:16:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5851 [2026-10-20 23:16:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5897 [2026-10-20 23:16:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1407 [2026-10-20 23:16:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8282 [2026-10-20 23:16:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1198 [2026-10-20 23:16:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2842 [2026-10-20 23:16:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2724 [2026-10-20 23:16:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5210 [2026-10-20 23:17:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.6717 [2026-10-20 23:17:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8402 [2026-10-20 23:17:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.5368 [2026-10-20 23:17:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1300 [2026-10-20 23:17:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3541 [2026-10-20 23:17:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1459 [2026-10-20 23:17:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3466 [2026-10-20 23:17:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5899 [2026-10-20 23:17:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1063 [2026-10-20 23:17:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4236 [2026-10-20 23:17:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2144 [2026-10-20 23:17:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1935 [2026-10-20 23:18:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4662 [2026-10-20 23:18:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1748 [2026-10-20 23:18:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4774 [2026-10-20 23:18:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2827 [2026-10-20 23:18:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8343 [2026-10-20 23:18:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5374 [2026-10-20 23:18:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4160 [2026-10-20 23:18:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5859 [2026-10-20 23:18:37][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,564 | MFU: 1.8% [2026-10-20 23:18:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2426 [2026-10-20 23:18:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9360 [2026-10-20 23:18:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8553 [2026-10-20 23:18:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2267 [2026-10-20 23:18:42][INFO][training.pretrain_engine] [Ph2|Step 717] loss=5.4908 | ppl=242.45 | aux=0.02507 | lr=3.00e-04 | gnorm=0.805 | tok/s=29,280 | MFU=7.0% | step=143248ms | tokens=0.759B | mem=4.2/32.7GB | cost=$376.79/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 14:24 (135.1h) | Phase3: SKIPPED | SFT: Oct 26 19:09 (139.8h) | DPO (Final): Oct 26 19:43 (140.4h) [2026-10-20 23:18:42][INFO][training.pretrain_engine] Starting global step 718 (accumulating 32 micro-batches)... [2026-10-20 23:18:42][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 23:18:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9354 [2026-10-20 23:18:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1331 [2026-10-20 23:18:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7444 [2026-10-20 23:19:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4500 [2026-10-20 23:19:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5542 [2026-10-20 23:19:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0286 [2026-10-20 23:19:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7927 [2026-10-20 23:19:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7432 [2026-10-20 23:19:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7294 [2026-10-20 23:19:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4274 [2026-10-20 23:19:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2397 [2026-10-20 23:19:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6072 [2026-10-20 23:19:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4799 [2026-10-20 23:19:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5862 [2026-10-20 23:19:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6936 [2026-10-20 23:19:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4107 [2026-10-20 23:20:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7362 [2026-10-20 23:20:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3645 [2026-10-20 23:20:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3295 [2026-10-20 23:20:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1581 [2026-10-20 23:20:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1734 [2026-10-20 23:20:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3880 [2026-10-20 23:20:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0384 [2026-10-20 23:20:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5083 [2026-10-20 23:20:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3836 [2026-10-20 23:20:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4235 [2026-10-20 23:20:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0072 [2026-10-20 23:20:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3447 [2026-10-20 23:21:01][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,535 | MFU: 1.8% [2026-10-20 23:21:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2730 [2026-10-20 23:21:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2702 [2026-10-20 23:21:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2999 [2026-10-20 23:21:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2842 [2026-10-20 23:21:05][INFO][training.pretrain_engine] [Ph2|Step 718] loss=5.4557 | ppl=234.08 | aux=0.02507 | lr=3.00e-04 | gnorm=0.891 | tok/s=29,279 | MFU=7.0% | step=143253ms | tokens=0.760B | mem=4.2/32.7GB | cost=$377.31/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 14:26 (135.1h) | Phase3: SKIPPED | SFT: Oct 26 19:11 (139.8h) | DPO (Final): Oct 26 19:45 (140.4h) [2026-10-20 23:21:05][INFO][training.pretrain_engine] Starting global step 719 (accumulating 32 micro-batches)... [2026-10-20 23:21:05][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 23:21:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7552 [2026-10-20 23:21:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5736 [2026-10-20 23:21:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4472 [2026-10-20 23:21:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4233 [2026-10-20 23:21:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1054 [2026-10-20 23:21:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3387 [2026-10-20 23:21:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1545 [2026-10-20 23:21:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9297 [2026-10-20 23:21:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0746 [2026-10-20 23:21:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4421 [2026-10-20 23:21:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1648 [2026-10-20 23:21:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4621 [2026-10-20 23:22:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5547 [2026-10-20 23:22:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7554 [2026-10-20 23:22:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.4703 [2026-10-20 23:22:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7107 [2026-10-20 23:22:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4245 [2026-10-20 23:22:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4559 [2026-10-20 23:22:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5668 [2026-10-20 23:22:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6703 [2026-10-20 23:22:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5568 [2026-10-20 23:22:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8387 [2026-10-20 23:22:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3240 [2026-10-20 23:22:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8204 [2026-10-20 23:23:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3399 [2026-10-20 23:23:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6388 [2026-10-20 23:23:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1986 [2026-10-20 23:23:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5380 [2026-10-20 23:23:24][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,533 | MFU: 1.8% [2026-10-20 23:23:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4968 [2026-10-20 23:23:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2215 [2026-10-20 23:23:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6352 [2026-10-20 23:23:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7817 [2026-10-20 23:23:29][INFO][training.pretrain_engine] [Ph2|Step 719] loss=5.5267 | ppl=251.32 | aux=0.02507 | lr=3.00e-04 | gnorm=0.840 | tok/s=29,278 | MFU=7.0% | step=143257ms | tokens=0.761B | mem=4.2/32.7GB | cost=$377.84/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 14:28 (135.1h) | Phase3: SKIPPED | SFT: Oct 26 19:13 (139.8h) | DPO (Final): Oct 26 19:47 (140.4h) [2026-10-20 23:23:29][INFO][training.pretrain_engine] Starting global step 720 (accumulating 32 micro-batches)... [2026-10-20 23:23:29][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 23:23:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3244 [2026-10-20 23:23:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0715 [2026-10-20 23:23:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6367 [2026-10-20 23:23:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4836 [2026-10-20 23:24:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3155 [2026-10-20 23:24:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7758 [2026-10-20 23:24:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7824 [2026-10-20 23:24:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8453 [2026-10-20 23:24:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7941 [2026-10-20 23:24:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8364 [2026-10-20 23:24:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3430 [2026-10-20 23:24:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8805 [2026-10-20 23:24:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1846 [2026-10-20 23:24:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2801 [2026-10-20 23:24:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5165 [2026-10-20 23:24:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1155 [2026-10-20 23:24:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1311 [2026-10-20 23:24:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2106 [2026-10-20 23:24:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1563 [2026-10-20 23:24:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4410 [2026-10-20 23:25:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6171 [2026-10-20 23:25:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2313 [2026-10-20 23:25:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5069 [2026-10-20 23:25:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2977 [2026-10-20 23:25:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4168 [2026-10-20 23:25:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1666 [2026-10-20 23:25:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7565 [2026-10-20 23:25:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6540 [2026-10-20 23:25:47][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,580 | MFU: 1.8% [2026-10-20 23:25:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5909 [2026-10-20 23:25:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5438 [2026-10-20 23:25:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4462 [2026-10-20 23:25:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4275 [2026-10-20 23:25:52][INFO][training.pretrain_engine] [Ph2|Step 720] loss=5.4717 | ppl=237.86 | aux=0.02508 | lr=3.00e-04 | gnorm=0.805 | tok/s=29,280 | MFU=7.0% | step=143250ms | tokens=0.762B | mem=4.2/32.7GB | cost=$378.36/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 14:30 (135.1h) | Phase3: SKIPPED | SFT: Oct 26 19:14 (139.8h) | DPO (Final): Oct 26 19:49 (140.4h) [2026-10-20 23:25:52][INFO][training.pretrain_engine] Starting global step 721 (accumulating 32 micro-batches)... [2026-10-20 23:25:52][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 23:26:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6762 [2026-10-20 23:26:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7904 [2026-10-20 23:26:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5491 [2026-10-20 23:26:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8230 [2026-10-20 23:26:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5348 [2026-10-20 23:26:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3870 [2026-10-20 23:26:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6726 [2026-10-20 23:26:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5656 [2026-10-20 23:26:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6257 [2026-10-20 23:26:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4010 [2026-10-20 23:26:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4533 [2026-10-20 23:26:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6032 [2026-10-20 23:27:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3135 [2026-10-20 23:27:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3709 [2026-10-20 23:27:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2831 [2026-10-20 23:27:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6365 [2026-10-20 23:27:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4904 [2026-10-20 23:27:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7652 [2026-10-20 23:27:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9559 [2026-10-20 23:27:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5612 [2026-10-20 23:27:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8401 [2026-10-20 23:27:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8194 [2026-10-20 23:27:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1812 [2026-10-20 23:27:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7416 [2026-10-20 23:27:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7650 [2026-10-20 23:27:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6448 [2026-10-20 23:27:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5844 [2026-10-20 23:27:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9655 [2026-10-20 23:28:10][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,602 | MFU: 1.8% [2026-10-20 23:28:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2319 [2026-10-20 23:28:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4798 [2026-10-20 23:28:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1539 [2026-10-20 23:28:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2793 [2026-10-20 23:28:15][INFO][training.pretrain_engine] [Ph2|Step 721] loss=5.4930 | ppl=242.99 | aux=0.02508 | lr=3.00e-04 | gnorm=1.031 | tok/s=29,280 | MFU=7.0% | step=143249ms | tokens=0.763B | mem=4.2/32.7GB | cost=$378.88/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 14:32 (135.1h) | Phase3: SKIPPED | SFT: Oct 26 19:16 (139.8h) | DPO (Final): Oct 26 19:50 (140.4h) [2026-10-20 23:28:15][INFO][training.pretrain_engine] Starting global step 722 (accumulating 32 micro-batches)... [2026-10-20 23:28:15][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 23:28:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1823 [2026-10-20 23:28:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7818 [2026-10-20 23:28:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4841 [2026-10-20 23:28:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2952 [2026-10-20 23:28:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2892 [2026-10-20 23:28:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7787 [2026-10-20 23:28:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7466 [2026-10-20 23:28:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8678 [2026-10-20 23:29:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5173 [2026-10-20 23:29:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7010 [2026-10-20 23:29:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5121 [2026-10-20 23:29:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7237 [2026-10-20 23:29:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3920 [2026-10-20 23:29:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3513 [2026-10-20 23:29:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0805 [2026-10-20 23:29:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7285 [2026-10-20 23:29:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4983 [2026-10-20 23:29:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6259 [2026-10-20 23:29:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0835 [2026-10-20 23:29:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2779 [2026-10-20 23:30:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4770 [2026-10-20 23:30:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9357 [2026-10-20 23:30:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8195 [2026-10-20 23:30:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9281 [2026-10-20 23:30:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0680 [2026-10-20 23:30:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8021 [2026-10-20 23:30:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6873 [2026-10-20 23:30:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3291 [2026-10-20 23:30:33][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,608 | MFU: 1.8% [2026-10-20 23:30:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4728 [2026-10-20 23:30:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4532 [2026-10-20 23:30:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2655 [2026-10-20 23:30:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2740 [2026-10-20 23:30:37][INFO][training.pretrain_engine] [Ph2|Step 722] loss=5.5015 | ppl=245.07 | aux=0.02508 | lr=3.00e-04 | gnorm=1.172 | tok/s=29,281 | MFU=7.0% | step=143246ms | tokens=0.764B | mem=4.2/32.7GB | cost=$379.40/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 14:33 (135.0h) | Phase3: SKIPPED | SFT: Oct 26 19:18 (139.8h) | DPO (Final): Oct 26 19:52 (140.4h) [2026-10-20 23:30:41][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step722 | Tokens: 0.764B | Spent: $379.41 | Remaining: $20.59 [2026-10-20 23:30:41][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step722 [2026-10-20 23:30:41][INFO][training.pretrain_engine] Starting global step 723 (accumulating 32 micro-batches)... [2026-10-20 23:30:41][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 23:30:41][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 23:30:41][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 23:30:41][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 23:30:41][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step722 to Phase-Technologies/forge-500m-pretrain (path: phase2_step722)... [2026-10-20 23:30:41][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 23:30:50][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 23:30:50][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step722 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 23:30:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7382 [2026-10-20 23:30:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1486 [2026-10-20 23:30:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6196 [2026-10-20 23:30:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5795 [2026-10-20 23:31:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7996 [2026-10-20 23:31:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2168 [2026-10-20 23:31:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.9677 [2026-10-20 23:31:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6191 [2026-10-20 23:31:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5890 [2026-10-20 23:31:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9608 [2026-10-20 23:31:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0888 [2026-10-20 23:31:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6619 [2026-10-20 23:31:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9111 [2026-10-20 23:31:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6625 [2026-10-20 23:31:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5404 [2026-10-20 23:31:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6624 [2026-10-20 23:32:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4395 [2026-10-20 23:32:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5955 [2026-10-20 23:32:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6129 [2026-10-20 23:32:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8449 [2026-10-20 23:32:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5867 [2026-10-20 23:32:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4610 [2026-10-20 23:32:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5635 [2026-10-20 23:32:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5292 [2026-10-20 23:32:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3708 [2026-10-20 23:32:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7217 [2026-10-20 23:32:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8697 [2026-10-20 23:32:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6639 [2026-10-20 23:32:58][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,623 | MFU: 1.8% [2026-10-20 23:33:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9372 [2026-10-20 23:33:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8025 [2026-10-20 23:33:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5932 [2026-10-20 23:33:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4858 [2026-10-20 23:33:03][INFO][training.pretrain_engine] [Ph2|Step 723] loss=5.6348 | ppl=279.99 | aux=0.02508 | lr=3.00e-04 | gnorm=0.914 | tok/s=29,283 | MFU=7.0% | step=143234ms | tokens=0.765B | mem=4.2/32.7GB | cost=$379.93/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 14:34 (135.0h) | Phase3: SKIPPED | SFT: Oct 26 19:19 (139.8h) | DPO (Final): Oct 26 19:53 (140.3h) [2026-10-20 23:33:03][INFO][training.pretrain_engine] Starting global step 724 (accumulating 32 micro-batches)... [2026-10-20 23:33:03][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 23:33:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4175 [2026-10-20 23:33:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0155 [2026-10-20 23:33:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1003 [2026-10-20 23:33:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8681 [2026-10-20 23:33:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7407 [2026-10-20 23:33:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1529 [2026-10-20 23:33:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7017 [2026-10-20 23:33:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1572 [2026-10-20 23:33:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7882 [2026-10-20 23:33:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1946 [2026-10-20 23:33:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1679 [2026-10-20 23:33:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6212 [2026-10-20 23:34:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3256 [2026-10-20 23:34:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1980 [2026-10-20 23:34:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2243 [2026-10-20 23:34:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5715 [2026-10-20 23:34:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2043 [2026-10-20 23:34:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5148 [2026-10-20 23:34:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8578 [2026-10-20 23:34:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8599 [2026-10-20 23:34:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5932 [2026-10-20 23:34:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8556 [2026-10-20 23:34:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5421 [2026-10-20 23:34:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7067 [2026-10-20 23:35:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3253 [2026-10-20 23:35:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9241 [2026-10-20 23:35:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5331 [2026-10-20 23:35:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9422 [2026-10-20 23:35:21][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,576 | MFU: 1.8% [2026-10-20 23:35:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5652 [2026-10-20 23:35:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7771 [2026-10-20 23:35:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5450 [2026-10-20 23:35:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1973 [2026-10-20 23:35:26][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $380.46 of $400.00 [2026-10-20 23:35:26][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $380.46 of $400.00 [2026-10-20 23:35:26][INFO][training.pretrain_engine] [Ph2|Step 724] loss=5.4604 | ppl=235.18 | aux=0.02508 | lr=3.00e-04 | gnorm=0.820 | tok/s=29,284 | MFU=7.0% | step=143230ms | tokens=0.767B | mem=4.2/32.7GB | cost=$380.46/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 14:36 (135.0h) | Phase3: SKIPPED | SFT: Oct 26 19:20 (139.8h) | DPO (Final): Oct 26 19:55 (140.3h) [2026-10-20 23:35:26][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $380.46 of $400.00 [2026-10-20 23:35:26][INFO][training.pretrain_engine] Starting global step 725 (accumulating 32 micro-batches)... [2026-10-20 23:35:26][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $380.46 of $400.00 [2026-10-20 23:35:26][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 23:35:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4419 [2026-10-20 23:35:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4682 [2026-10-20 23:35:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3287 [2026-10-20 23:35:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4508 [2026-10-20 23:36:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4929 [2026-10-20 23:36:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6143 [2026-10-20 23:36:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8725 [2026-10-20 23:36:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4379 [2026-10-20 23:36:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4167 [2026-10-20 23:36:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1266 [2026-10-20 23:36:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2807 [2026-10-20 23:36:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9343 [2026-10-20 23:36:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3051 [2026-10-20 23:36:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1257 [2026-10-20 23:36:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4472 [2026-10-20 23:36:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8564 [2026-10-20 23:36:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8653 [2026-10-20 23:36:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6478 [2026-10-20 23:36:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1865 [2026-10-20 23:36:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6618 [2026-10-20 23:37:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9854 [2026-10-20 23:37:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3937 [2026-10-20 23:37:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6311 [2026-10-20 23:37:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4778 [2026-10-20 23:37:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3614 [2026-10-20 23:37:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6305 [2026-10-20 23:37:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9238 [2026-10-20 23:37:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6145 [2026-10-20 23:37:44][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,561 | MFU: 1.8% [2026-10-20 23:37:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4846 [2026-10-20 23:37:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5204 [2026-10-20 23:37:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1194 [2026-10-20 23:37:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8291 [2026-10-20 23:37:49][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $380.98 of $400.00 [2026-10-20 23:37:49][INFO][training.pretrain_engine] [Ph2|Step 725] loss=5.4867 | ppl=241.45 | aux=0.02507 | lr=3.00e-04 | gnorm=0.711 | tok/s=29,284 | MFU=7.0% | step=143229ms | tokens=0.768B | mem=4.2/32.7GB | cost=$380.98/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 14:38 (135.0h) | Phase3: SKIPPED | SFT: Oct 26 19:22 (139.7h) | DPO (Final): Oct 26 19:56 (140.3h) [2026-10-20 23:37:49][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $380.98 of $400.00 [2026-10-20 23:37:49][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $380.98 of $400.00 [2026-10-20 23:37:49][INFO][training.pretrain_engine] Starting global step 726 (accumulating 32 micro-batches)... [2026-10-20 23:37:49][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $380.98 of $400.00 [2026-10-20 23:37:49][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 23:38:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6073 [2026-10-20 23:38:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5056 [2026-10-20 23:38:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0585 [2026-10-20 23:38:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8277 [2026-10-20 23:38:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8617 [2026-10-20 23:38:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6572 [2026-10-20 23:38:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1828 [2026-10-20 23:38:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6652 [2026-10-20 23:38:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9833 [2026-10-20 23:38:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7665 [2026-10-20 23:38:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6232 [2026-10-20 23:38:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8785 [2026-10-20 23:39:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1572 [2026-10-20 23:39:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4767 [2026-10-20 23:39:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3384 [2026-10-20 23:39:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4900 [2026-10-20 23:39:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4530 [2026-10-20 23:39:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5627 [2026-10-20 23:39:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8592 [2026-10-20 23:39:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6193 [2026-10-20 23:39:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8485 [2026-10-20 23:39:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6495 [2026-10-20 23:39:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7519 [2026-10-20 23:39:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2270 [2026-10-20 23:39:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3995 [2026-10-20 23:39:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3530 [2026-10-20 23:39:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3083 [2026-10-20 23:39:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2822 [2026-10-20 23:40:07][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,575 | MFU: 1.8% [2026-10-20 23:40:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6332 [2026-10-20 23:40:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1949 [2026-10-20 23:40:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9703 [2026-10-20 23:40:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1684 [2026-10-20 23:40:12][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $381.50 of $400.00 [2026-10-20 23:40:12][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $381.50 of $400.00 [2026-10-20 23:40:12][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $381.50 of $400.00 [2026-10-20 23:40:12][INFO][training.pretrain_engine] [Ph2|Step 726] loss=5.4593 | ppl=234.94 | aux=0.02506 | lr=3.00e-04 | gnorm=0.637 | tok/s=29,284 | MFU=7.0% | step=143227ms | tokens=0.769B | mem=4.2/32.7GB | cost=$381.50/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 14:39 (135.0h) | Phase3: SKIPPED | SFT: Oct 26 19:24 (139.7h) | DPO (Final): Oct 26 19:58 (140.3h) [2026-10-20 23:40:12][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $381.50 of $400.00 [2026-10-20 23:40:12][INFO][training.pretrain_engine] Starting global step 727 (accumulating 32 micro-batches)... [2026-10-20 23:40:12][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 23:40:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3167 [2026-10-20 23:40:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6948 [2026-10-20 23:40:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1344 [2026-10-20 23:40:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9913 [2026-10-20 23:40:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8258 [2026-10-20 23:40:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5690 [2026-10-20 23:40:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5547 [2026-10-20 23:40:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2090 [2026-10-20 23:41:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3538 [2026-10-20 23:41:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5712 [2026-10-20 23:41:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2414 [2026-10-20 23:41:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3603 [2026-10-20 23:41:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1599 [2026-10-20 23:41:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3817 [2026-10-20 23:41:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4049 [2026-10-20 23:41:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8045 [2026-10-20 23:41:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6941 [2026-10-20 23:41:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6054 [2026-10-20 23:41:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4688 [2026-10-20 23:41:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5013 [2026-10-20 23:41:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6201 [2026-10-20 23:41:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4100 [2026-10-20 23:41:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5286 [2026-10-20 23:41:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4656 [2026-10-20 23:42:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4614 [2026-10-20 23:42:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3862 [2026-10-20 23:42:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4827 [2026-10-20 23:42:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5510 [2026-10-20 23:42:30][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,600 | MFU: 1.8% [2026-10-20 23:42:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2045 [2026-10-20 23:42:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.1120 [2026-10-20 23:42:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2712 [2026-10-20 23:42:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8147 [2026-10-20 23:42:35][INFO][training.pretrain_engine] [Ph2|Step 727] loss=5.5219 | ppl=250.12 | aux=0.02507 | lr=3.00e-04 | gnorm=0.715 | tok/s=29,284 | MFU=7.0% | step=143228ms | tokens=0.770B | mem=4.2/32.7GB | cost=$382.03/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 14:41 (135.0h) | Phase3: SKIPPED | SFT: Oct 26 19:26 (139.7h) | DPO (Final): Oct 26 20:00 (140.3h) [2026-10-20 23:42:35][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $382.03 of $400.00 [2026-10-20 23:42:35][INFO][training.pretrain_engine] Starting global step 728 (accumulating 32 micro-batches)... [2026-10-20 23:42:35][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 23:42:35][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $382.03 of $400.00 [2026-10-20 23:42:35][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $382.03 of $400.00 [2026-10-20 23:42:35][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $382.03 of $400.00 [2026-10-20 23:42:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4672 [2026-10-20 23:42:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3573 [2026-10-20 23:42:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2599 [2026-10-20 23:42:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1529 [2026-10-20 23:43:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0176 [2026-10-20 23:43:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9521 [2026-10-20 23:43:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8218 [2026-10-20 23:43:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2120 [2026-10-20 23:43:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4578 [2026-10-20 23:43:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4127 [2026-10-20 23:43:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5854 [2026-10-20 23:43:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5901 [2026-10-20 23:43:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8714 [2026-10-20 23:43:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5497 [2026-10-20 23:43:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6207 [2026-10-20 23:43:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2394 [2026-10-20 23:44:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5831 [2026-10-20 23:44:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2608 [2026-10-20 23:44:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2051 [2026-10-20 23:44:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5673 [2026-10-20 23:44:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3643 [2026-10-20 23:44:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8723 [2026-10-20 23:44:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6098 [2026-10-20 23:44:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6278 [2026-10-20 23:44:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2130 [2026-10-20 23:44:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6331 [2026-10-20 23:44:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4927 [2026-10-20 23:44:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5820 [2026-10-20 23:44:53][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,605 | MFU: 1.8% [2026-10-20 23:44:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8596 [2026-10-20 23:44:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6402 [2026-10-20 23:44:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8105 [2026-10-20 23:44:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9816 [2026-10-20 23:44:58][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $382.55 of $400.00 [2026-10-20 23:44:58][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $382.55 of $400.00 [2026-10-20 23:44:58][INFO][training.pretrain_engine] [Ph2|Step 728] loss=5.4383 | ppl=230.05 | aux=0.02507 | lr=3.00e-04 | gnorm=0.734 | tok/s=29,285 | MFU=7.0% | step=143225ms | tokens=0.771B | mem=4.2/32.7GB | cost=$382.55/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 14:43 (135.0h) | Phase3: SKIPPED | SFT: Oct 26 19:27 (139.7h) | DPO (Final): Oct 26 20:01 (140.3h) [2026-10-20 23:44:58][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $382.55 of $400.00 [2026-10-20 23:44:58][INFO][training.pretrain_engine] Starting global step 729 (accumulating 32 micro-batches)... [2026-10-20 23:44:58][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $382.55 of $400.00 [2026-10-20 23:44:58][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 23:45:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4695 [2026-10-20 23:45:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2467 [2026-10-20 23:45:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1964 [2026-10-20 23:45:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5770 [2026-10-20 23:45:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2660 [2026-10-20 23:45:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3217 [2026-10-20 23:45:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3877 [2026-10-20 23:45:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2570 [2026-10-20 23:45:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3264 [2026-10-20 23:45:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9349 [2026-10-20 23:45:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6411 [2026-10-20 23:45:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9051 [2026-10-20 23:46:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9628 [2026-10-20 23:46:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7201 [2026-10-20 23:46:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0375 [2026-10-20 23:46:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.3687 [2026-10-20 23:46:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4208 [2026-10-20 23:46:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7752 [2026-10-20 23:46:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1429 [2026-10-20 23:46:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4357 [2026-10-20 23:46:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3467 [2026-10-20 23:46:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4708 [2026-10-20 23:46:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3718 [2026-10-20 23:46:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2502 [2026-10-20 23:47:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1966 [2026-10-20 23:47:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1309 [2026-10-20 23:47:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8298 [2026-10-20 23:47:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9431 [2026-10-20 23:47:17][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,577 | MFU: 1.8% [2026-10-20 23:47:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4100 [2026-10-20 23:47:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8221 [2026-10-20 23:47:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4722 [2026-10-20 23:47:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6518 [2026-10-20 23:47:21][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $383.07 of $400.00 [2026-10-20 23:47:21][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $383.07 of $400.00 [2026-10-20 23:47:21][INFO][training.pretrain_engine] [Ph2|Step 729] loss=5.4888 | ppl=241.96 | aux=0.02508 | lr=3.00e-04 | gnorm=0.688 | tok/s=29,285 | MFU=7.0% | step=143223ms | tokens=0.772B | mem=4.2/32.7GB | cost=$383.07/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 14:44 (135.0h) | Phase3: SKIPPED | SFT: Oct 26 19:29 (139.7h) | DPO (Final): Oct 26 20:03 (140.3h) [2026-10-20 23:47:21][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $383.07 of $400.00 [2026-10-20 23:47:21][INFO][training.pretrain_engine] Starting global step 730 (accumulating 32 micro-batches)... [2026-10-20 23:47:21][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $383.07 of $400.00 [2026-10-20 23:47:21][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 23:47:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0309 [2026-10-20 23:47:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3082 [2026-10-20 23:47:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5008 [2026-10-20 23:47:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3894 [2026-10-20 23:47:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5636 [2026-10-20 23:47:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4845 [2026-10-20 23:47:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6822 [2026-10-20 23:47:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4249 [2026-10-20 23:48:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8825 [2026-10-20 23:48:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8053 [2026-10-20 23:48:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7046 [2026-10-20 23:48:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6927 [2026-10-20 23:48:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7596 [2026-10-20 23:48:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5637 [2026-10-20 23:48:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4077 [2026-10-20 23:48:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4996 [2026-10-20 23:48:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0835 [2026-10-20 23:48:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8202 [2026-10-20 23:48:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4178 [2026-10-20 23:48:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3406 [2026-10-20 23:49:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3760 [2026-10-20 23:49:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2329 [2026-10-20 23:49:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8211 [2026-10-20 23:49:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0004 [2026-10-20 23:49:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6591 [2026-10-20 23:49:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5750 [2026-10-20 23:49:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5748 [2026-10-20 23:49:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3173 [2026-10-20 23:49:40][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,552 | MFU: 1.8% [2026-10-20 23:49:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7184 [2026-10-20 23:49:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6822 [2026-10-20 23:49:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0643 [2026-10-20 23:49:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3907 [2026-10-20 23:49:45][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $383.60 of $400.00 [2026-10-20 23:49:45][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $383.60 of $400.00 [2026-10-20 23:49:45][INFO][training.pretrain_engine] [Ph2|Step 730] loss=5.4268 | ppl=227.42 | aux=0.02508 | lr=3.00e-04 | gnorm=0.645 | tok/s=29,285 | MFU=7.0% | step=143224ms | tokens=0.773B | mem=4.2/32.7GB | cost=$383.60/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 14:46 (135.0h) | Phase3: SKIPPED | SFT: Oct 26 19:31 (139.7h) | DPO (Final): Oct 26 20:05 (140.3h) [2026-10-20 23:49:45][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $383.60 of $400.00 [2026-10-20 23:49:45][INFO][training.pretrain_engine] Starting global step 731 (accumulating 32 micro-batches)... [2026-10-20 23:49:45][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $383.60 of $400.00 [2026-10-20 23:49:45][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 23:50:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4799 [2026-10-20 23:50:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7249 [2026-10-20 23:50:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8058 [2026-10-20 23:50:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6554 [2026-10-20 23:50:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5779 [2026-10-20 23:50:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5905 [2026-10-20 23:50:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5719 [2026-10-20 23:50:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4930 [2026-10-20 23:50:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4809 [2026-10-20 23:50:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9709 [2026-10-20 23:50:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8702 [2026-10-20 23:50:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7741 [2026-10-20 23:50:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5585 [2026-10-20 23:50:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3410 [2026-10-20 23:50:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6212 [2026-10-20 23:50:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8795 [2026-10-20 23:51:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4948 [2026-10-20 23:51:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7163 [2026-10-20 23:51:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9690 [2026-10-20 23:51:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6850 [2026-10-20 23:51:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4110 [2026-10-20 23:51:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8985 [2026-10-20 23:51:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7284 [2026-10-20 23:51:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4093 [2026-10-20 23:51:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5424 [2026-10-20 23:51:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1953 [2026-10-20 23:51:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2640 [2026-10-20 23:51:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8188 [2026-10-20 23:52:03][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,563 | MFU: 1.8% [2026-10-20 23:52:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5992 [2026-10-20 23:52:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5495 [2026-10-20 23:52:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7233 [2026-10-20 23:52:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5209 [2026-10-20 23:52:08][INFO][training.pretrain_engine] [Ph2|Step 731] loss=5.4702 | ppl=237.51 | aux=0.02507 | lr=3.00e-04 | gnorm=0.719 | tok/s=29,285 | MFU=7.0% | step=143224ms | tokens=0.774B | mem=4.2/32.7GB | cost=$384.12/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 14:48 (134.9h) | Phase3: SKIPPED | SFT: Oct 26 19:33 (139.7h) | DPO (Final): Oct 26 20:07 (140.3h) [2026-10-20 23:52:08][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $384.12 of $400.00 [2026-10-20 23:52:08][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $384.12 of $400.00 [2026-10-20 23:52:08][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $384.12 of $400.00 [2026-10-20 23:52:11][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step731 | Tokens: 0.774B | Spent: $384.13 | Remaining: $15.87 [2026-10-20 23:52:11][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step731 [2026-10-20 23:52:11][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $384.13 of $400.00 [2026-10-20 23:52:11][INFO][training.pretrain_engine] Starting global step 732 (accumulating 32 micro-batches)... [2026-10-20 23:52:11][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 23:52:11][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-20 23:52:11][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-20 23:52:12][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-20 23:52:12][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step731 to Phase-Technologies/forge-500m-pretrain (path: phase2_step731)... [2026-10-20 23:52:12][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-20 23:52:21][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-20 23:52:21][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step731 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-20 23:52:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6261 [2026-10-20 23:52:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7436 [2026-10-20 23:52:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8711 [2026-10-20 23:52:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7024 [2026-10-20 23:52:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8375 [2026-10-20 23:52:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6920 [2026-10-20 23:52:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4957 [2026-10-20 23:52:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7648 [2026-10-20 23:53:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7866 [2026-10-20 23:53:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5051 [2026-10-20 23:53:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3469 [2026-10-20 23:53:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6993 [2026-10-20 23:53:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2190 [2026-10-20 23:53:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5871 [2026-10-20 23:53:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3000 [2026-10-20 23:53:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3655 [2026-10-20 23:53:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0242 [2026-10-20 23:53:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4066 [2026-10-20 23:53:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2013 [2026-10-20 23:53:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5657 [2026-10-20 23:53:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1523 [2026-10-20 23:53:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9559 [2026-10-20 23:53:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1783 [2026-10-20 23:53:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8947 [2026-10-20 23:54:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1290 [2026-10-20 23:54:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6164 [2026-10-20 23:54:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2698 [2026-10-20 23:54:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9486 [2026-10-20 23:54:30][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,563 | MFU: 1.8% [2026-10-20 23:54:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5781 [2026-10-20 23:54:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4620 [2026-10-20 23:54:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5669 [2026-10-20 23:54:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0090 [2026-10-20 23:54:35][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $384.66 of $400.00 [2026-10-20 23:54:35][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $384.66 of $400.00 [2026-10-20 23:54:35][INFO][training.pretrain_engine] [Ph2|Step 732] loss=5.4641 | ppl=236.07 | aux=0.02507 | lr=3.00e-04 | gnorm=0.617 | tok/s=29,286 | MFU=7.0% | step=143219ms | tokens=0.775B | mem=4.2/32.7GB | cost=$384.66/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 14:50 (134.9h) | Phase3: SKIPPED | SFT: Oct 26 19:34 (139.7h) | DPO (Final): Oct 26 20:08 (140.2h) [2026-10-20 23:54:35][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $384.66 of $400.00 [2026-10-20 23:54:35][INFO][training.pretrain_engine] Starting global step 733 (accumulating 32 micro-batches)... [2026-10-20 23:54:35][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $384.66 of $400.00 [2026-10-20 23:54:35][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 23:54:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3069 [2026-10-20 23:54:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0492 [2026-10-20 23:54:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2286 [2026-10-20 23:54:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2192 [2026-10-20 23:55:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5263 [2026-10-20 23:55:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2827 [2026-10-20 23:55:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2122 [2026-10-20 23:55:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1642 [2026-10-20 23:55:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0616 [2026-10-20 23:55:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2422 [2026-10-20 23:55:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2149 [2026-10-20 23:55:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2152 [2026-10-20 23:55:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4073 [2026-10-20 23:55:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1596 [2026-10-20 23:55:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0221 [2026-10-20 23:55:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3256 [2026-10-20 23:56:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5471 [2026-10-20 23:56:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8306 [2026-10-20 23:56:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7824 [2026-10-20 23:56:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7442 [2026-10-20 23:56:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6802 [2026-10-20 23:56:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9447 [2026-10-20 23:56:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3162 [2026-10-20 23:56:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4283 [2026-10-20 23:56:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6480 [2026-10-20 23:56:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4042 [2026-10-20 23:56:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8355 [2026-10-20 23:56:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7362 [2026-10-20 23:56:53][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,570 | MFU: 1.8% [2026-10-20 23:56:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9785 [2026-10-20 23:56:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2637 [2026-10-20 23:56:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4125 [2026-10-20 23:56:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4134 [2026-10-20 23:56:58][INFO][training.pretrain_engine] [Ph2|Step 733] loss=5.4665 | ppl=236.62 | aux=0.02507 | lr=3.00e-04 | gnorm=0.688 | tok/s=29,287 | MFU=7.0% | step=143216ms | tokens=0.776B | mem=4.2/32.7GB | cost=$385.18/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 14:51 (134.9h) | Phase3: SKIPPED | SFT: Oct 26 19:36 (139.7h) | DPO (Final): Oct 26 20:10 (140.2h) [2026-10-20 23:56:58][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $385.18 of $400.00 [2026-10-20 23:56:58][INFO][training.pretrain_engine] Starting global step 734 (accumulating 32 micro-batches)... [2026-10-20 23:56:58][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $385.18 of $400.00 [2026-10-20 23:56:58][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $385.18 of $400.00 [2026-10-20 23:56:58][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 23:56:58][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $385.18 of $400.00 [2026-10-20 23:57:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9712 [2026-10-20 23:57:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2565 [2026-10-20 23:57:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1381 [2026-10-20 23:57:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4714 [2026-10-20 23:57:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5639 [2026-10-20 23:57:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0893 [2026-10-20 23:57:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1519 [2026-10-20 23:57:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2119 [2026-10-20 23:57:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6613 [2026-10-20 23:57:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6394 [2026-10-20 23:57:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5474 [2026-10-20 23:57:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3664 [2026-10-20 23:58:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7143 [2026-10-20 23:58:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3472 [2026-10-20 23:58:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8580 [2026-10-20 23:58:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6288 [2026-10-20 23:58:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5235 [2026-10-20 23:58:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5923 [2026-10-20 23:58:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4343 [2026-10-20 23:58:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8183 [2026-10-20 23:58:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9418 [2026-10-20 23:58:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0820 [2026-10-20 23:58:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2831 [2026-10-20 23:58:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3772 [2026-10-20 23:59:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9024 [2026-10-20 23:59:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9235 [2026-10-20 23:59:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0421 [2026-10-20 23:59:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6165 [2026-10-20 23:59:16][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,598 | MFU: 1.8% [2026-10-20 23:59:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9921 [2026-10-20 23:59:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9962 [2026-10-20 23:59:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3191 [2026-10-20 23:59:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7307 [2026-10-20 23:59:22][INFO][training.pretrain_engine] [Ph2|Step 734] loss=5.3777 | ppl=216.53 | aux=0.02508 | lr=3.00e-04 | gnorm=0.590 | tok/s=29,286 | MFU=7.0% | step=143220ms | tokens=0.777B | mem=4.2/32.7GB | cost=$385.70/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 14:53 (134.9h) | Phase3: SKIPPED | SFT: Oct 26 19:38 (139.6h) | DPO (Final): Oct 26 20:12 (140.2h) [2026-10-20 23:59:22][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $385.70 of $400.00 [2026-10-20 23:59:22][INFO][training.pretrain_engine] Starting global step 735 (accumulating 32 micro-batches)... [2026-10-20 23:59:22][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-20 23:59:22][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $385.70 of $400.00 [2026-10-20 23:59:22][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $385.70 of $400.00 [2026-10-20 23:59:22][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $385.70 of $400.00 [2026-10-20 23:59:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.9825 [2026-10-20 23:59:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6341 [2026-10-20 23:59:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1824 [2026-10-20 23:59:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3237 [2026-10-20 23:59:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4377 [2026-10-20 23:59:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0981 [2026-10-20 23:59:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5002 [2026-10-20 23:59:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6785 [2026-10-21 00:00:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9517 [2026-10-21 00:00:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6360 [2026-10-21 00:00:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1773 [2026-10-21 00:00:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2273 [2026-10-21 00:00:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7512 [2026-10-21 00:00:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5582 [2026-10-21 00:00:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7502 [2026-10-21 00:00:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6625 [2026-10-21 00:00:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9538 [2026-10-21 00:00:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3620 [2026-10-21 00:00:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7532 [2026-10-21 00:00:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2407 [2026-10-21 00:01:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4920 [2026-10-21 00:01:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0676 [2026-10-21 00:01:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4173 [2026-10-21 00:01:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3589 [2026-10-21 00:01:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6061 [2026-10-21 00:01:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0115 [2026-10-21 00:01:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7737 [2026-10-21 00:01:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9491 [2026-10-21 00:01:40][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,599 | MFU: 1.8% [2026-10-21 00:01:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0760 [2026-10-21 00:01:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9071 [2026-10-21 00:01:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8070 [2026-10-21 00:01:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3046 [2026-10-21 00:01:45][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $386.23 of $400.00 [2026-10-21 00:01:45][INFO][training.pretrain_engine] [Ph2|Step 735] loss=5.3326 | ppl=206.98 | aux=0.02508 | lr=3.00e-04 | gnorm=0.586 | tok/s=29,287 | MFU=7.0% | step=143211ms | tokens=0.778B | mem=4.2/32.7GB | cost=$386.23/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 14:55 (134.9h) | Phase3: SKIPPED | SFT: Oct 26 19:39 (139.6h) | DPO (Final): Oct 26 20:13 (140.2h) [2026-10-21 00:01:45][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $386.23 of $400.00 [2026-10-21 00:01:45][INFO][training.pretrain_engine] Starting global step 736 (accumulating 32 micro-batches)... [2026-10-21 00:01:45][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $386.23 of $400.00 [2026-10-21 00:01:45][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 00:01:45][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $386.23 of $400.00 [2026-10-21 00:02:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3843 [2026-10-21 00:02:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8799 [2026-10-21 00:02:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2545 [2026-10-21 00:02:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0456 [2026-10-21 00:02:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1064 [2026-10-21 00:02:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2210 [2026-10-21 00:02:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1986 [2026-10-21 00:02:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2413 [2026-10-21 00:02:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1798 [2026-10-21 00:02:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3308 [2026-10-21 00:02:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7250 [2026-10-21 00:02:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5162 [2026-10-21 00:02:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7851 [2026-10-21 00:02:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6228 [2026-10-21 00:02:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9463 [2026-10-21 00:02:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9388 [2026-10-21 00:03:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4321 [2026-10-21 00:03:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3322 [2026-10-21 00:03:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5368 [2026-10-21 00:03:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1607 [2026-10-21 00:03:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4858 [2026-10-21 00:03:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3006 [2026-10-21 00:03:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2396 [2026-10-21 00:03:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4118 [2026-10-21 00:03:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3992 [2026-10-21 00:03:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4449 [2026-10-21 00:03:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5832 [2026-10-21 00:03:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2509 [2026-10-21 00:04:03][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,604 | MFU: 1.8% [2026-10-21 00:04:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5685 [2026-10-21 00:04:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7653 [2026-10-21 00:04:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8208 [2026-10-21 00:04:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6418 [2026-10-21 00:04:07][INFO][training.pretrain_engine] [Ph2|Step 736] loss=5.3631 | ppl=213.39 | aux=0.02508 | lr=3.00e-04 | gnorm=0.609 | tok/s=29,288 | MFU=7.0% | step=143208ms | tokens=0.779B | mem=4.2/32.7GB | cost=$386.75/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 14:56 (134.9h) | Phase3: SKIPPED | SFT: Oct 26 19:41 (139.6h) | DPO (Final): Oct 26 20:15 (140.2h) [2026-10-21 00:04:07][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $386.75 of $400.00 [2026-10-21 00:04:07][INFO][training.pretrain_engine] Starting global step 737 (accumulating 32 micro-batches)... [2026-10-21 00:04:07][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $386.75 of $400.00 [2026-10-21 00:04:07][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 00:04:07][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $386.75 of $400.00 [2026-10-21 00:04:07][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $386.75 of $400.00 [2026-10-21 00:04:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4976 [2026-10-21 00:04:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3640 [2026-10-21 00:04:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4221 [2026-10-21 00:04:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4282 [2026-10-21 00:04:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1938 [2026-10-21 00:04:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6838 [2026-10-21 00:04:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2711 [2026-10-21 00:04:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5105 [2026-10-21 00:05:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8377 [2026-10-21 00:05:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3868 [2026-10-21 00:05:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2135 [2026-10-21 00:05:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4302 [2026-10-21 00:05:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5446 [2026-10-21 00:05:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6840 [2026-10-21 00:05:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6020 [2026-10-21 00:05:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5230 [2026-10-21 00:05:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3671 [2026-10-21 00:05:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0952 [2026-10-21 00:05:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4008 [2026-10-21 00:05:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1126 [2026-10-21 00:05:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6614 [2026-10-21 00:05:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9147 [2026-10-21 00:05:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0910 [2026-10-21 00:05:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2291 [2026-10-21 00:06:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9289 [2026-10-21 00:06:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6898 [2026-10-21 00:06:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7329 [2026-10-21 00:06:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7163 [2026-10-21 00:06:25][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,596 | MFU: 1.8% [2026-10-21 00:06:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7789 [2026-10-21 00:06:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0231 [2026-10-21 00:06:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2237 [2026-10-21 00:06:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8954 [2026-10-21 00:06:31][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $387.27 of $400.00 [2026-10-21 00:06:31][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $387.27 of $400.00 [2026-10-21 00:06:31][INFO][training.pretrain_engine] [Ph2|Step 737] loss=5.4412 | ppl=230.71 | aux=0.02508 | lr=3.00e-04 | gnorm=0.504 | tok/s=29,288 | MFU=7.0% | step=143210ms | tokens=0.780B | mem=4.2/32.7GB | cost=$387.27/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 14:58 (134.9h) | Phase3: SKIPPED | SFT: Oct 26 19:43 (139.6h) | DPO (Final): Oct 26 20:17 (140.2h) [2026-10-21 00:06:31][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $387.27 of $400.00 [2026-10-21 00:06:31][INFO][training.pretrain_engine] Starting global step 738 (accumulating 32 micro-batches)... [2026-10-21 00:06:31][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $387.27 of $400.00 [2026-10-21 00:06:31][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 00:06:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3121 [2026-10-21 00:06:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7562 [2026-10-21 00:06:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6534 [2026-10-21 00:06:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6149 [2026-10-21 00:07:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3761 [2026-10-21 00:07:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5251 [2026-10-21 00:07:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4694 [2026-10-21 00:07:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5383 [2026-10-21 00:07:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5984 [2026-10-21 00:07:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0102 [2026-10-21 00:07:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5867 [2026-10-21 00:07:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2540 [2026-10-21 00:07:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6348 [2026-10-21 00:07:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.6169 [2026-10-21 00:07:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9047 [2026-10-21 00:07:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6610 [2026-10-21 00:08:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3554 [2026-10-21 00:08:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2177 [2026-10-21 00:08:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5846 [2026-10-21 00:08:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4447 [2026-10-21 00:08:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1068 [2026-10-21 00:08:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9484 [2026-10-21 00:08:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8256 [2026-10-21 00:08:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2807 [2026-10-21 00:08:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3259 [2026-10-21 00:08:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2126 [2026-10-21 00:08:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8235 [2026-10-21 00:08:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2426 [2026-10-21 00:08:49][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,577 | MFU: 1.8% [2026-10-21 00:08:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6144 [2026-10-21 00:08:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3390 [2026-10-21 00:08:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5157 [2026-10-21 00:08:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5777 [2026-10-21 00:08:54][INFO][training.pretrain_engine] [Ph2|Step 738] loss=5.4590 | ppl=234.87 | aux=0.02509 | lr=3.00e-04 | gnorm=0.574 | tok/s=29,287 | MFU=7.0% | step=143212ms | tokens=0.781B | mem=4.2/32.7GB | cost=$387.80/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 15:00 (134.9h) | Phase3: SKIPPED | SFT: Oct 26 19:44 (139.6h) | DPO (Final): Oct 26 20:19 (140.2h) [2026-10-21 00:08:54][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $387.80 of $400.00 [2026-10-21 00:08:54][INFO][training.pretrain_engine] Starting global step 739 (accumulating 32 micro-batches)... [2026-10-21 00:08:54][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $387.80 of $400.00 [2026-10-21 00:08:54][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $387.80 of $400.00 [2026-10-21 00:08:54][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $387.80 of $400.00 [2026-10-21 00:08:54][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 00:09:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5151 [2026-10-21 00:09:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4575 [2026-10-21 00:09:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2456 [2026-10-21 00:09:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5933 [2026-10-21 00:09:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0944 [2026-10-21 00:09:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1082 [2026-10-21 00:09:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3286 [2026-10-21 00:09:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4767 [2026-10-21 00:09:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2272 [2026-10-21 00:09:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0833 [2026-10-21 00:09:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6294 [2026-10-21 00:09:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6252 [2026-10-21 00:10:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4010 [2026-10-21 00:10:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6398 [2026-10-21 00:10:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4233 [2026-10-21 00:10:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3684 [2026-10-21 00:10:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3061 [2026-10-21 00:10:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1395 [2026-10-21 00:10:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1952 [2026-10-21 00:10:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3966 [2026-10-21 00:10:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5193 [2026-10-21 00:10:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8685 [2026-10-21 00:10:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0980 [2026-10-21 00:10:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9162 [2026-10-21 00:10:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2019 [2026-10-21 00:10:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4635 [2026-10-21 00:10:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9110 [2026-10-21 00:10:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0842 [2026-10-21 00:11:12][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,576 | MFU: 1.8% [2026-10-21 00:11:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6268 [2026-10-21 00:11:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3893 [2026-10-21 00:11:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5064 [2026-10-21 00:11:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4278 [2026-10-21 00:11:17][INFO][training.pretrain_engine] [Ph2|Step 739] loss=5.3916 | ppl=219.56 | aux=0.02509 | lr=3.00e-04 | gnorm=0.486 | tok/s=29,288 | MFU=7.0% | step=143208ms | tokens=0.782B | mem=4.2/32.7GB | cost=$388.32/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 15:01 (134.8h) | Phase3: SKIPPED | SFT: Oct 26 19:46 (139.6h) | DPO (Final): Oct 26 20:20 (140.2h) [2026-10-21 00:11:17][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $388.32 of $400.00 [2026-10-21 00:11:17][INFO][training.pretrain_engine] Starting global step 740 (accumulating 32 micro-batches)... [2026-10-21 00:11:17][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $388.32 of $400.00 [2026-10-21 00:11:17][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 00:11:17][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $388.32 of $400.00 [2026-10-21 00:11:17][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $388.32 of $400.00 [2026-10-21 00:11:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0492 [2026-10-21 00:11:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6904 [2026-10-21 00:11:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0516 [2026-10-21 00:11:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9844 [2026-10-21 00:11:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0895 [2026-10-21 00:11:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6388 [2026-10-21 00:11:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5538 [2026-10-21 00:11:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4794 [2026-10-21 00:12:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8062 [2026-10-21 00:12:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4490 [2026-10-21 00:12:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8059 [2026-10-21 00:12:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6017 [2026-10-21 00:12:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4412 [2026-10-21 00:12:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3254 [2026-10-21 00:12:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6149 [2026-10-21 00:12:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6888 [2026-10-21 00:12:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7290 [2026-10-21 00:12:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8591 [2026-10-21 00:12:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7743 [2026-10-21 00:12:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9700 [2026-10-21 00:13:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4146 [2026-10-21 00:13:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7451 [2026-10-21 00:13:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9366 [2026-10-21 00:13:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0166 [2026-10-21 00:13:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7401 [2026-10-21 00:13:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5105 [2026-10-21 00:13:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0044 [2026-10-21 00:13:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7100 [2026-10-21 00:13:35][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,570 | MFU: 1.8% [2026-10-21 00:13:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3969 [2026-10-21 00:13:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7975 [2026-10-21 00:13:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6426 [2026-10-21 00:13:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9104 [2026-10-21 00:13:40][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $388.84 of $400.00 [2026-10-21 00:13:40][INFO][training.pretrain_engine] [Ph2|Step 740] loss=5.4181 | ppl=225.45 | aux=0.02506 | lr=3.00e-04 | gnorm=0.582 | tok/s=29,288 | MFU=7.0% | step=143207ms | tokens=0.783B | mem=4.2/32.7GB | cost=$388.84/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 15:03 (134.8h) | Phase3: SKIPPED | SFT: Oct 26 19:48 (139.6h) | DPO (Final): Oct 26 20:22 (140.1h) [2026-10-21 00:13:40][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $388.84 of $400.00 [2026-10-21 00:13:40][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $388.84 of $400.00 [2026-10-21 00:13:43][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step740 | Tokens: 0.783B | Spent: $388.85 | Remaining: $11.15 [2026-10-21 00:13:43][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step740 [2026-10-21 00:13:43][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $388.85 of $400.00 [2026-10-21 00:13:43][INFO][training.pretrain_engine] Starting global step 741 (accumulating 32 micro-batches)... [2026-10-21 00:13:43][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 00:13:43][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 00:13:43][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 00:13:43][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 00:13:43][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step740 to Phase-Technologies/forge-500m-pretrain (path: phase2_step740)... [2026-10-21 00:13:44][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 00:13:53][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 00:13:53][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step740 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 00:13:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7140 [2026-10-21 00:13:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1515 [2026-10-21 00:13:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6418 [2026-10-21 00:14:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6743 [2026-10-21 00:14:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9649 [2026-10-21 00:14:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1028 [2026-10-21 00:14:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0928 [2026-10-21 00:14:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3676 [2026-10-21 00:14:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6794 [2026-10-21 00:14:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5262 [2026-10-21 00:14:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7246 [2026-10-21 00:14:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6970 [2026-10-21 00:14:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0787 [2026-10-21 00:14:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2593 [2026-10-21 00:14:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6906 [2026-10-21 00:14:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1970 [2026-10-21 00:15:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8440 [2026-10-21 00:15:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5270 [2026-10-21 00:15:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8483 [2026-10-21 00:15:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5674 [2026-10-21 00:15:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4195 [2026-10-21 00:15:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4114 [2026-10-21 00:15:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2644 [2026-10-21 00:15:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4622 [2026-10-21 00:15:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4873 [2026-10-21 00:15:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6864 [2026-10-21 00:15:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6730 [2026-10-21 00:15:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2807 [2026-10-21 00:16:02][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,575 | MFU: 1.8% [2026-10-21 00:16:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3894 [2026-10-21 00:16:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8694 [2026-10-21 00:16:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8400 [2026-10-21 00:16:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2578 [2026-10-21 00:16:06][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $389.38 of $400.00 [2026-10-21 00:16:06][INFO][training.pretrain_engine] [Ph2|Step 741] loss=5.3639 | ppl=213.55 | aux=0.02506 | lr=3.00e-04 | gnorm=0.531 | tok/s=29,288 | MFU=7.0% | step=143207ms | tokens=0.784B | mem=4.2/32.7GB | cost=$389.38/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 15:05 (134.8h) | Phase3: SKIPPED | SFT: Oct 26 19:50 (139.6h) | DPO (Final): Oct 26 20:24 (140.1h) [2026-10-21 00:16:06][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $389.38 of $400.00 [2026-10-21 00:16:06][INFO][training.pretrain_engine] Starting global step 742 (accumulating 32 micro-batches)... [2026-10-21 00:16:06][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $389.38 of $400.00 [2026-10-21 00:16:06][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $389.38 of $400.00 [2026-10-21 00:16:06][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 00:16:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9528 [2026-10-21 00:16:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0860 [2026-10-21 00:16:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3377 [2026-10-21 00:16:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4846 [2026-10-21 00:16:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9179 [2026-10-21 00:16:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2611 [2026-10-21 00:16:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.6661 [2026-10-21 00:16:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5715 [2026-10-21 00:17:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0619 [2026-10-21 00:17:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9368 [2026-10-21 00:17:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7988 [2026-10-21 00:17:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3293 [2026-10-21 00:17:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6171 [2026-10-21 00:17:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6412 [2026-10-21 00:17:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8590 [2026-10-21 00:17:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7266 [2026-10-21 00:17:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2719 [2026-10-21 00:17:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1665 [2026-10-21 00:17:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4711 [2026-10-21 00:17:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0605 [2026-10-21 00:17:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4211 [2026-10-21 00:17:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1241 [2026-10-21 00:17:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2289 [2026-10-21 00:17:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5512 [2026-10-21 00:18:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9266 [2026-10-21 00:18:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5197 [2026-10-21 00:18:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2893 [2026-10-21 00:18:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8683 [2026-10-21 00:18:24][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,578 | MFU: 1.8% [2026-10-21 00:18:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4007 [2026-10-21 00:18:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5691 [2026-10-21 00:18:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2922 [2026-10-21 00:18:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4875 [2026-10-21 00:18:29][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $389.90 of $400.00 [2026-10-21 00:18:29][INFO][training.pretrain_engine] [Ph2|Step 742] loss=5.4525 | ppl=233.35 | aux=0.02508 | lr=3.00e-04 | gnorm=0.672 | tok/s=29,289 | MFU=7.0% | step=143202ms | tokens=0.785B | mem=4.2/32.7GB | cost=$389.90/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 15:07 (134.8h) | Phase3: SKIPPED | SFT: Oct 26 19:51 (139.6h) | DPO (Final): Oct 26 20:25 (140.1h) [2026-10-21 00:18:29][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $389.90 of $400.00 [2026-10-21 00:18:29][INFO][training.pretrain_engine] Starting global step 743 (accumulating 32 micro-batches)... [2026-10-21 00:18:29][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 00:18:29][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $389.90 of $400.00 [2026-10-21 00:18:29][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $389.90 of $400.00 [2026-10-21 00:18:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7831 [2026-10-21 00:18:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0442 [2026-10-21 00:18:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3110 [2026-10-21 00:18:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4589 [2026-10-21 00:19:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3462 [2026-10-21 00:19:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0988 [2026-10-21 00:19:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.9323 [2026-10-21 00:19:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1483 [2026-10-21 00:19:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0400 [2026-10-21 00:19:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4918 [2026-10-21 00:19:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.4651 [2026-10-21 00:19:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7233 [2026-10-21 00:19:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9741 [2026-10-21 00:19:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8275 [2026-10-21 00:19:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1272 [2026-10-21 00:19:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0732 [2026-10-21 00:19:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0090 [2026-10-21 00:19:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2717 [2026-10-21 00:19:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2078 [2026-10-21 00:19:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2090 [2026-10-21 00:20:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7660 [2026-10-21 00:20:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9070 [2026-10-21 00:20:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6180 [2026-10-21 00:20:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7296 [2026-10-21 00:20:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6206 [2026-10-21 00:20:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6806 [2026-10-21 00:20:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6119 [2026-10-21 00:20:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8809 [2026-10-21 00:20:48][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,545 | MFU: 1.8% [2026-10-21 00:20:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3407 [2026-10-21 00:20:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3109 [2026-10-21 00:20:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6278 [2026-10-21 00:20:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5954 [2026-10-21 00:20:53][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $390.42 of $400.00 [2026-10-21 00:20:53][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $390.42 of $400.00 [2026-10-21 00:20:53][INFO][training.pretrain_engine] [Ph2|Step 743] loss=5.4202 | ppl=225.92 | aux=0.02508 | lr=3.00e-04 | gnorm=0.727 | tok/s=29,289 | MFU=7.0% | step=143202ms | tokens=0.786B | mem=4.2/32.7GB | cost=$390.42/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 15:08 (134.8h) | Phase3: SKIPPED | SFT: Oct 26 19:53 (139.5h) | DPO (Final): Oct 26 20:27 (140.1h) [2026-10-21 00:20:53][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $390.42 of $400.00 [2026-10-21 00:20:53][INFO][training.pretrain_engine] Starting global step 744 (accumulating 32 micro-batches)... [2026-10-21 00:20:53][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $390.42 of $400.00 [2026-10-21 00:20:53][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 00:21:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1614 [2026-10-21 00:21:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8181 [2026-10-21 00:21:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2409 [2026-10-21 00:21:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6618 [2026-10-21 00:21:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8440 [2026-10-21 00:21:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8919 [2026-10-21 00:21:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0169 [2026-10-21 00:21:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7962 [2026-10-21 00:21:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8427 [2026-10-21 00:21:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5901 [2026-10-21 00:21:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3962 [2026-10-21 00:21:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0756 [2026-10-21 00:22:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5798 [2026-10-21 00:22:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0844 [2026-10-21 00:22:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4351 [2026-10-21 00:22:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2556 [2026-10-21 00:22:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0915 [2026-10-21 00:22:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4088 [2026-10-21 00:22:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8629 [2026-10-21 00:22:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8497 [2026-10-21 00:22:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1152 [2026-10-21 00:22:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6303 [2026-10-21 00:22:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6777 [2026-10-21 00:22:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1220 [2026-10-21 00:22:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0490 [2026-10-21 00:22:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1560 [2026-10-21 00:22:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9490 [2026-10-21 00:22:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6654 [2026-10-21 00:23:11][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,583 | MFU: 1.8% [2026-10-21 00:23:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5502 [2026-10-21 00:23:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4041 [2026-10-21 00:23:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3165 [2026-10-21 00:23:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4158 [2026-10-21 00:23:16][INFO][training.pretrain_engine] [Ph2|Step 744] loss=5.4705 | ppl=237.57 | aux=0.02508 | lr=3.00e-04 | gnorm=0.742 | tok/s=29,290 | MFU=7.0% | step=143199ms | tokens=0.787B | mem=4.2/32.7GB | cost=$390.95/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 15:10 (134.8h) | Phase3: SKIPPED | SFT: Oct 26 19:55 (139.5h) | DPO (Final): Oct 26 20:29 (140.1h) [2026-10-21 00:23:16][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $390.95 of $400.00 [2026-10-21 00:23:16][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $390.95 of $400.00 [2026-10-21 00:23:16][INFO][training.pretrain_engine] Starting global step 745 (accumulating 32 micro-batches)... [2026-10-21 00:23:16][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $390.95 of $400.00 [2026-10-21 00:23:16][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 00:23:16][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $390.95 of $400.00 [2026-10-21 00:23:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8284 [2026-10-21 00:23:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6499 [2026-10-21 00:23:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2420 [2026-10-21 00:23:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7734 [2026-10-21 00:23:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5785 [2026-10-21 00:23:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6371 [2026-10-21 00:23:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1575 [2026-10-21 00:23:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8476 [2026-10-21 00:24:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7619 [2026-10-21 00:24:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3007 [2026-10-21 00:24:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6651 [2026-10-21 00:24:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7526 [2026-10-21 00:24:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6746 [2026-10-21 00:24:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1437 [2026-10-21 00:24:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8878 [2026-10-21 00:24:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1657 [2026-10-21 00:24:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9269 [2026-10-21 00:24:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2000 [2026-10-21 00:24:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7227 [2026-10-21 00:24:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5994 [2026-10-21 00:25:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6124 [2026-10-21 00:25:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6601 [2026-10-21 00:25:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3903 [2026-10-21 00:25:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5465 [2026-10-21 00:25:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5237 [2026-10-21 00:25:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9632 [2026-10-21 00:25:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1842 [2026-10-21 00:25:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9043 [2026-10-21 00:25:34][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,570 | MFU: 1.8% [2026-10-21 00:25:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0279 [2026-10-21 00:25:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4008 [2026-10-21 00:25:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6191 [2026-10-21 00:25:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5271 [2026-10-21 00:25:39][INFO][training.pretrain_engine] [Ph2|Step 745] loss=5.5424 | ppl=255.30 | aux=0.02509 | lr=3.00e-04 | gnorm=0.707 | tok/s=29,289 | MFU=7.0% | step=143203ms | tokens=0.789B | mem=4.2/32.7GB | cost=$391.47/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 15:12 (134.8h) | Phase3: SKIPPED | SFT: Oct 26 19:57 (139.5h) | DPO (Final): Oct 26 20:31 (140.1h) [2026-10-21 00:25:39][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $391.47 of $400.00 [2026-10-21 00:25:39][INFO][training.pretrain_engine] Starting global step 746 (accumulating 32 micro-batches)... [2026-10-21 00:25:39][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 00:25:39][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $391.47 of $400.00 [2026-10-21 00:25:39][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $391.47 of $400.00 [2026-10-21 00:25:39][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $391.47 of $400.00 [2026-10-21 00:25:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2061 [2026-10-21 00:25:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2496 [2026-10-21 00:25:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2201 [2026-10-21 00:25:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2978 [2026-10-21 00:26:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9946 [2026-10-21 00:26:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5838 [2026-10-21 00:26:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9363 [2026-10-21 00:26:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4798 [2026-10-21 00:26:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6767 [2026-10-21 00:26:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6947 [2026-10-21 00:26:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0176 [2026-10-21 00:26:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4509 [2026-10-21 00:26:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4752 [2026-10-21 00:26:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2917 [2026-10-21 00:26:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0393 [2026-10-21 00:26:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6702 [2026-10-21 00:27:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7000 [2026-10-21 00:27:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2330 [2026-10-21 00:27:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3275 [2026-10-21 00:27:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0023 [2026-10-21 00:27:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8165 [2026-10-21 00:27:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9274 [2026-10-21 00:27:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0166 [2026-10-21 00:27:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8053 [2026-10-21 00:27:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5749 [2026-10-21 00:27:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5672 [2026-10-21 00:27:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8972 [2026-10-21 00:27:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4958 [2026-10-21 00:27:57][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,601 | MFU: 1.8% [2026-10-21 00:28:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9188 [2026-10-21 00:28:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3099 [2026-10-21 00:28:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0850 [2026-10-21 00:28:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4225 [2026-10-21 00:28:02][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $391.99 of $400.00 [2026-10-21 00:28:02][INFO][training.pretrain_engine] [Ph2|Step 746] loss=5.2410 | ppl=188.86 | aux=0.02509 | lr=3.00e-04 | gnorm=0.660 | tok/s=29,289 | MFU=7.0% | step=143202ms | tokens=0.790B | mem=4.2/32.7GB | cost=$391.99/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 15:14 (134.8h) | Phase3: SKIPPED | SFT: Oct 26 19:58 (139.5h) | DPO (Final): Oct 26 20:32 (140.1h) [2026-10-21 00:28:02][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $391.99 of $400.00 [2026-10-21 00:28:02][INFO][training.pretrain_engine] Starting global step 747 (accumulating 32 micro-batches)... [2026-10-21 00:28:02][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $391.99 of $400.00 [2026-10-21 00:28:02][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 00:28:02][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $391.99 of $400.00 [2026-10-21 00:28:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8101 [2026-10-21 00:28:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3857 [2026-10-21 00:28:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2881 [2026-10-21 00:28:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2436 [2026-10-21 00:28:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1067 [2026-10-21 00:28:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2246 [2026-10-21 00:28:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3758 [2026-10-21 00:28:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1824 [2026-10-21 00:28:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7526 [2026-10-21 00:28:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4541 [2026-10-21 00:28:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0239 [2026-10-21 00:28:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9008 [2026-10-21 00:29:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9928 [2026-10-21 00:29:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6260 [2026-10-21 00:29:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6058 [2026-10-21 00:29:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3675 [2026-10-21 00:29:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9643 [2026-10-21 00:29:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7537 [2026-10-21 00:29:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3003 [2026-10-21 00:29:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6281 [2026-10-21 00:29:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5200 [2026-10-21 00:29:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7369 [2026-10-21 00:29:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4316 [2026-10-21 00:29:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4352 [2026-10-21 00:30:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2104 [2026-10-21 00:30:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5389 [2026-10-21 00:30:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3932 [2026-10-21 00:30:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6587 [2026-10-21 00:30:21][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,567 | MFU: 1.8% [2026-10-21 00:30:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1130 [2026-10-21 00:30:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0858 [2026-10-21 00:30:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5514 [2026-10-21 00:30:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9757 [2026-10-21 00:30:25][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $392.52 of $400.00 [2026-10-21 00:30:25][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $392.52 of $400.00 [2026-10-21 00:30:25][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $392.52 of $400.00 [2026-10-21 00:30:25][INFO][training.pretrain_engine] [Ph2|Step 747] loss=5.3701 | ppl=214.89 | aux=0.02508 | lr=3.00e-04 | gnorm=0.805 | tok/s=29,289 | MFU=7.0% | step=143202ms | tokens=0.791B | mem=4.2/32.7GB | cost=$392.52/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 15:16 (134.8h) | Phase3: SKIPPED | SFT: Oct 26 20:00 (139.5h) | DPO (Final): Oct 26 20:34 (140.1h) [2026-10-21 00:30:25][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $392.52 of $400.00 [2026-10-21 00:30:25][INFO][training.pretrain_engine] Starting global step 748 (accumulating 32 micro-batches)... [2026-10-21 00:30:25][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 00:30:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0244 [2026-10-21 00:30:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4581 [2026-10-21 00:30:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3660 [2026-10-21 00:30:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5015 [2026-10-21 00:31:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1086 [2026-10-21 00:31:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1960 [2026-10-21 00:31:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3549 [2026-10-21 00:31:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4956 [2026-10-21 00:31:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7079 [2026-10-21 00:31:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4676 [2026-10-21 00:31:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7219 [2026-10-21 00:31:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0801 [2026-10-21 00:31:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3383 [2026-10-21 00:31:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2705 [2026-10-21 00:31:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1036 [2026-10-21 00:31:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5370 [2026-10-21 00:31:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2481 [2026-10-21 00:31:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9470 [2026-10-21 00:31:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9183 [2026-10-21 00:31:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8260 [2026-10-21 00:32:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0070 [2026-10-21 00:32:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5282 [2026-10-21 00:32:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3508 [2026-10-21 00:32:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4786 [2026-10-21 00:32:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1062 [2026-10-21 00:32:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1453 [2026-10-21 00:32:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1802 [2026-10-21 00:32:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1134 [2026-10-21 00:32:44][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,556 | MFU: 1.8% [2026-10-21 00:32:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5695 [2026-10-21 00:32:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2634 [2026-10-21 00:32:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2259 [2026-10-21 00:32:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5005 [2026-10-21 00:32:49][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $393.04 of $400.00 [2026-10-21 00:32:49][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $393.04 of $400.00 [2026-10-21 00:32:49][INFO][training.pretrain_engine] [Ph2|Step 748] loss=5.3938 | ppl=220.05 | aux=0.02509 | lr=3.00e-04 | gnorm=1.086 | tok/s=29,288 | MFU=7.0% | step=143207ms | tokens=0.792B | mem=4.2/32.7GB | cost=$393.04/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 15:18 (134.8h) | Phase3: SKIPPED | SFT: Oct 26 20:02 (139.5h) | DPO (Final): Oct 26 20:36 (140.1h) [2026-10-21 00:32:49][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $393.04 of $400.00 [2026-10-21 00:32:49][INFO][training.pretrain_engine] Starting global step 749 (accumulating 32 micro-batches)... [2026-10-21 00:32:49][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 00:32:49][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $393.04 of $400.00 [2026-10-21 00:33:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3648 [2026-10-21 00:33:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7248 [2026-10-21 00:33:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7645 [2026-10-21 00:33:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1444 [2026-10-21 00:33:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6994 [2026-10-21 00:33:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3068 [2026-10-21 00:33:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5249 [2026-10-21 00:33:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2694 [2026-10-21 00:33:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3536 [2026-10-21 00:33:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4389 [2026-10-21 00:33:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1336 [2026-10-21 00:33:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2449 [2026-10-21 00:34:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5951 [2026-10-21 00:34:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6164 [2026-10-21 00:34:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9670 [2026-10-21 00:34:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4344 [2026-10-21 00:34:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9535 [2026-10-21 00:34:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8654 [2026-10-21 00:34:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4887 [2026-10-21 00:34:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8005 [2026-10-21 00:34:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3820 [2026-10-21 00:34:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7437 [2026-10-21 00:34:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2796 [2026-10-21 00:34:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5353 [2026-10-21 00:34:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2927 [2026-10-21 00:34:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5707 [2026-10-21 00:34:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0400 [2026-10-21 00:34:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3553 [2026-10-21 00:35:08][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,548 | MFU: 1.8% [2026-10-21 00:35:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5734 [2026-10-21 00:35:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8295 [2026-10-21 00:35:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5278 [2026-10-21 00:35:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5964 [2026-10-21 00:35:12][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $393.57 of $400.00 [2026-10-21 00:35:12][INFO][training.pretrain_engine] [Ph2|Step 749] loss=5.4525 | ppl=233.34 | aux=0.02509 | lr=3.00e-04 | gnorm=1.141 | tok/s=29,287 | MFU=7.0% | step=143213ms | tokens=0.793B | mem=4.2/32.7GB | cost=$393.57/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 15:20 (134.8h) | Phase3: SKIPPED | SFT: Oct 26 20:04 (139.5h) | DPO (Final): Oct 26 20:38 (140.1h) [2026-10-21 00:35:12][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $393.57 of $400.00 [2026-10-21 00:35:12][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $393.57 of $400.00 [2026-10-21 00:35:15][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step749 | Tokens: 0.793B | Spent: $393.58 | Remaining: $6.42 [2026-10-21 00:35:15][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step749 [2026-10-21 00:35:15][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $393.58 of $400.00 [2026-10-21 00:35:15][INFO][training.pretrain_engine] Starting global step 750 (accumulating 32 micro-batches)... [2026-10-21 00:35:15][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 00:35:16][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 00:35:16][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 00:35:16][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 00:35:16][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step749 to Phase-Technologies/forge-500m-pretrain (path: phase2_step749)... [2026-10-21 00:35:16][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 00:35:26][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 00:35:26][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step749 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 00:35:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9505 [2026-10-21 00:35:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8166 [2026-10-21 00:35:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0011 [2026-10-21 00:35:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8267 [2026-10-21 00:35:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2366 [2026-10-21 00:35:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9910 [2026-10-21 00:35:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9439 [2026-10-21 00:35:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4231 [2026-10-21 00:36:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2079 [2026-10-21 00:36:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1773 [2026-10-21 00:36:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8568 [2026-10-21 00:36:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4289 [2026-10-21 00:36:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4531 [2026-10-21 00:36:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9986 [2026-10-21 00:36:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1523 [2026-10-21 00:36:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3399 [2026-10-21 00:36:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8401 [2026-10-21 00:36:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1949 [2026-10-21 00:36:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2037 [2026-10-21 00:36:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4375 [2026-10-21 00:36:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8972 [2026-10-21 00:36:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0001 [2026-10-21 00:36:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6166 [2026-10-21 00:37:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4859 [2026-10-21 00:37:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7313 [2026-10-21 00:37:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6511 [2026-10-21 00:37:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0466 [2026-10-21 00:37:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9318 [2026-10-21 00:37:34][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,547 | MFU: 1.8% [2026-10-21 00:37:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8580 [2026-10-21 00:37:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5267 [2026-10-21 00:37:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6677 [2026-10-21 00:37:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4407 [2026-10-21 00:37:39][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $394.10 of $400.00 [2026-10-21 00:37:39][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $394.10 of $400.00 [2026-10-21 00:37:39][INFO][training.pretrain_engine] [Ph2|Step 750] loss=5.4244 | ppl=226.88 | aux=0.02507 | lr=3.00e-04 | gnorm=0.855 | tok/s=29,287 | MFU=7.0% | step=143216ms | tokens=0.794B | mem=4.2/32.7GB | cost=$394.10/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 15:22 (134.7h) | Phase3: SKIPPED | SFT: Oct 26 20:06 (139.5h) | DPO (Final): Oct 26 20:40 (140.1h) [2026-10-21 00:37:39][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $394.10 of $400.00 [2026-10-21 00:37:39][INFO][training.pretrain_engine] Starting global step 751 (accumulating 32 micro-batches)... [2026-10-21 00:37:39][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 00:37:39][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $394.10 of $400.00 [2026-10-21 00:37:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5402 [2026-10-21 00:37:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4974 [2026-10-21 00:37:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4155 [2026-10-21 00:37:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2782 [2026-10-21 00:38:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5562 [2026-10-21 00:38:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2550 [2026-10-21 00:38:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4281 [2026-10-21 00:38:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1256 [2026-10-21 00:38:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4244 [2026-10-21 00:38:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4035 [2026-10-21 00:38:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2709 [2026-10-21 00:38:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6061 [2026-10-21 00:38:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3306 [2026-10-21 00:38:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0356 [2026-10-21 00:38:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6029 [2026-10-21 00:38:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7199 [2026-10-21 00:39:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8711 [2026-10-21 00:39:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0164 [2026-10-21 00:39:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5464 [2026-10-21 00:39:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4162 [2026-10-21 00:39:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2073 [2026-10-21 00:39:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2205 [2026-10-21 00:39:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3891 [2026-10-21 00:39:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0738 [2026-10-21 00:39:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5426 [2026-10-21 00:39:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7752 [2026-10-21 00:39:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4578 [2026-10-21 00:39:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8711 [2026-10-21 00:39:58][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,566 | MFU: 1.8% [2026-10-21 00:40:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1849 [2026-10-21 00:40:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4274 [2026-10-21 00:40:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5281 [2026-10-21 00:40:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5968 [2026-10-21 00:40:03][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $394.63 of $400.00 [2026-10-21 00:40:03][INFO][training.pretrain_engine] [Ph2|Step 751] loss=5.3861 | ppl=218.36 | aux=0.02507 | lr=3.00e-04 | gnorm=0.855 | tok/s=29,286 | MFU=7.0% | step=143219ms | tokens=0.795B | mem=4.2/32.7GB | cost=$394.63/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 15:24 (134.7h) | Phase3: SKIPPED | SFT: Oct 26 20:08 (139.5h) | DPO (Final): Oct 26 20:42 (140.0h) [2026-10-21 00:40:03][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $394.63 of $400.00 [2026-10-21 00:40:03][INFO][training.pretrain_engine] Starting global step 752 (accumulating 32 micro-batches)... [2026-10-21 00:40:03][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $394.63 of $400.00 [2026-10-21 00:40:03][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $394.63 of $400.00 [2026-10-21 00:40:03][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 00:40:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7186 [2026-10-21 00:40:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9157 [2026-10-21 00:40:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6190 [2026-10-21 00:40:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6553 [2026-10-21 00:40:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5417 [2026-10-21 00:40:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3424 [2026-10-21 00:40:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0957 [2026-10-21 00:40:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5488 [2026-10-21 00:40:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6365 [2026-10-21 00:40:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9924 [2026-10-21 00:40:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6045 [2026-10-21 00:40:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5516 [2026-10-21 00:41:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9748 [2026-10-21 00:41:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6470 [2026-10-21 00:41:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8810 [2026-10-21 00:41:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7978 [2026-10-21 00:41:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6021 [2026-10-21 00:41:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7643 [2026-10-21 00:41:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4571 [2026-10-21 00:41:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8101 [2026-10-21 00:41:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2217 [2026-10-21 00:41:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6097 [2026-10-21 00:41:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6509 [2026-10-21 00:41:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3830 [2026-10-21 00:42:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4735 [2026-10-21 00:42:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5517 [2026-10-21 00:42:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4698 [2026-10-21 00:42:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5854 [2026-10-21 00:42:21][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,598 | MFU: 1.8% [2026-10-21 00:42:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2688 [2026-10-21 00:42:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9473 [2026-10-21 00:42:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2084 [2026-10-21 00:42:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7548 [2026-10-21 00:42:26][INFO][training.pretrain_engine] [Ph2|Step 752] loss=5.4605 | ppl=235.22 | aux=0.02508 | lr=3.00e-04 | gnorm=0.758 | tok/s=29,287 | MFU=7.0% | step=143212ms | tokens=0.796B | mem=4.2/32.7GB | cost=$395.15/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 15:25 (134.7h) | Phase3: SKIPPED | SFT: Oct 26 20:10 (139.5h) | DPO (Final): Oct 26 20:44 (140.0h) [2026-10-21 00:42:26][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $395.15 of $400.00 [2026-10-21 00:42:26][INFO][training.pretrain_engine] Starting global step 753 (accumulating 32 micro-batches)... [2026-10-21 00:42:26][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $395.15 of $400.00 [2026-10-21 00:42:26][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 00:42:26][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $395.15 of $400.00 [2026-10-21 00:42:26][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $395.15 of $400.00 [2026-10-21 00:42:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3343 [2026-10-21 00:42:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3603 [2026-10-21 00:42:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5205 [2026-10-21 00:42:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8094 [2026-10-21 00:43:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5746 [2026-10-21 00:43:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3738 [2026-10-21 00:43:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6034 [2026-10-21 00:43:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4422 [2026-10-21 00:43:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3088 [2026-10-21 00:43:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2054 [2026-10-21 00:43:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3730 [2026-10-21 00:43:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4423 [2026-10-21 00:43:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1867 [2026-10-21 00:43:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9803 [2026-10-21 00:43:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3923 [2026-10-21 00:43:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6205 [2026-10-21 00:43:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2086 [2026-10-21 00:43:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3866 [2026-10-21 00:43:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2408 [2026-10-21 00:43:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1802 [2026-10-21 00:44:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9027 [2026-10-21 00:44:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3650 [2026-10-21 00:44:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2168 [2026-10-21 00:44:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2740 [2026-10-21 00:44:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3851 [2026-10-21 00:44:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6508 [2026-10-21 00:44:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5184 [2026-10-21 00:44:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8029 [2026-10-21 00:44:44][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,582 | MFU: 1.8% [2026-10-21 00:44:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=3.9741 [2026-10-21 00:44:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6086 [2026-10-21 00:44:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3697 [2026-10-21 00:44:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.0505 [2026-10-21 00:44:50][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $395.68 of $400.00 [2026-10-21 00:44:50][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $395.68 of $400.00 [2026-10-21 00:44:50][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $395.68 of $400.00 [2026-10-21 00:44:50][INFO][training.pretrain_engine] [Ph2|Step 753] loss=5.3782 | ppl=216.63 | aux=0.02509 | lr=3.00e-04 | gnorm=0.715 | tok/s=29,285 | MFU=7.0% | step=143222ms | tokens=0.797B | mem=4.2/32.7GB | cost=$395.68/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 15:27 (134.7h) | Phase3: SKIPPED | SFT: Oct 26 20:12 (139.5h) | DPO (Final): Oct 26 20:46 (140.0h) [2026-10-21 00:44:50][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $395.68 of $400.00 [2026-10-21 00:44:50][INFO][training.pretrain_engine] Starting global step 754 (accumulating 32 micro-batches)... [2026-10-21 00:44:50][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 00:45:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1326 [2026-10-21 00:45:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4057 [2026-10-21 00:45:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2407 [2026-10-21 00:45:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1409 [2026-10-21 00:45:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0361 [2026-10-21 00:45:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2939 [2026-10-21 00:45:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1107 [2026-10-21 00:45:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1048 [2026-10-21 00:45:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0007 [2026-10-21 00:45:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5550 [2026-10-21 00:45:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2169 [2026-10-21 00:45:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4811 [2026-10-21 00:46:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4249 [2026-10-21 00:46:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3032 [2026-10-21 00:46:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2545 [2026-10-21 00:46:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7663 [2026-10-21 00:46:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2507 [2026-10-21 00:46:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9541 [2026-10-21 00:46:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5452 [2026-10-21 00:46:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7585 [2026-10-21 00:46:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2483 [2026-10-21 00:46:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7702 [2026-10-21 00:46:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3055 [2026-10-21 00:46:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3937 [2026-10-21 00:46:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4034 [2026-10-21 00:46:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7536 [2026-10-21 00:46:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9480 [2026-10-21 00:46:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6702 [2026-10-21 00:47:09][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,570 | MFU: 1.8% [2026-10-21 00:47:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1394 [2026-10-21 00:47:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5382 [2026-10-21 00:47:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9459 [2026-10-21 00:47:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1499 [2026-10-21 00:47:14][INFO][training.pretrain_engine] [Ph2|Step 754] loss=5.4562 | ppl=234.21 | aux=0.02509 | lr=3.00e-04 | gnorm=0.785 | tok/s=29,284 | MFU=7.0% | step=143231ms | tokens=0.798B | mem=4.2/32.7GB | cost=$396.21/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 15:30 (134.7h) | Phase3: SKIPPED | SFT: Oct 26 20:14 (139.5h) | DPO (Final): Oct 26 20:49 (140.0h) [2026-10-21 00:47:14][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $396.21 of $400.00 [2026-10-21 00:47:14][INFO][training.pretrain_engine] Starting global step 755 (accumulating 32 micro-batches)... [2026-10-21 00:47:14][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 00:47:14][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $396.21 of $400.00 [2026-10-21 00:47:14][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $396.21 of $400.00 [2026-10-21 00:47:14][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $396.21 of $400.00 [2026-10-21 00:47:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6629 [2026-10-21 00:47:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3311 [2026-10-21 00:47:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8291 [2026-10-21 00:47:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8623 [2026-10-21 00:47:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8539 [2026-10-21 00:47:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8634 [2026-10-21 00:47:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2953 [2026-10-21 00:47:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3920 [2026-10-21 00:48:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6712 [2026-10-21 00:48:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0042 [2026-10-21 00:48:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4793 [2026-10-21 00:48:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9753 [2026-10-21 00:48:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3780 [2026-10-21 00:48:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1101 [2026-10-21 00:48:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1932 [2026-10-21 00:48:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3452 [2026-10-21 00:48:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4231 [2026-10-21 00:48:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5163 [2026-10-21 00:48:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2342 [2026-10-21 00:48:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2768 [2026-10-21 00:49:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5919 [2026-10-21 00:49:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5929 [2026-10-21 00:49:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1542 [2026-10-21 00:49:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4546 [2026-10-21 00:49:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7494 [2026-10-21 00:49:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6191 [2026-10-21 00:49:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5369 [2026-10-21 00:49:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3671 [2026-10-21 00:49:33][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,562 | MFU: 1.8% [2026-10-21 00:49:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9976 [2026-10-21 00:49:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7489 [2026-10-21 00:49:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9150 [2026-10-21 00:49:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4758 [2026-10-21 00:49:38][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $396.73 of $400.00 [2026-10-21 00:49:38][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $396.73 of $400.00 [2026-10-21 00:49:38][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $396.73 of $400.00 [2026-10-21 00:49:38][INFO][training.pretrain_engine] [Ph2|Step 755] loss=5.4601 | ppl=235.12 | aux=0.02507 | lr=3.00e-04 | gnorm=0.680 | tok/s=29,284 | MFU=7.0% | step=143229ms | tokens=0.799B | mem=4.2/32.7GB | cost=$396.73/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 15:32 (134.7h) | Phase3: SKIPPED | SFT: Oct 26 20:16 (139.4h) | DPO (Final): Oct 26 20:50 (140.0h) [2026-10-21 00:49:38][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $396.73 of $400.00 [2026-10-21 00:49:38][INFO][training.pretrain_engine] Starting global step 756 (accumulating 32 micro-batches)... [2026-10-21 00:49:38][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 00:49:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1140 [2026-10-21 00:49:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7890 [2026-10-21 00:49:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9087 [2026-10-21 00:49:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9340 [2026-10-21 00:50:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4299 [2026-10-21 00:50:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4734 [2026-10-21 00:50:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2538 [2026-10-21 00:50:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5493 [2026-10-21 00:50:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2382 [2026-10-21 00:50:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1409 [2026-10-21 00:50:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3804 [2026-10-21 00:50:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5594 [2026-10-21 00:50:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6093 [2026-10-21 00:50:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3939 [2026-10-21 00:50:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9746 [2026-10-21 00:50:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3873 [2026-10-21 00:51:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3940 [2026-10-21 00:51:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0495 [2026-10-21 00:51:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4665 [2026-10-21 00:51:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1211 [2026-10-21 00:51:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6051 [2026-10-21 00:51:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9491 [2026-10-21 00:51:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1472 [2026-10-21 00:51:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7229 [2026-10-21 00:51:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2646 [2026-10-21 00:51:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9937 [2026-10-21 00:51:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3776 [2026-10-21 00:51:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1345 [2026-10-21 00:51:56][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,589 | MFU: 1.8% [2026-10-21 00:52:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3306 [2026-10-21 00:52:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3329 [2026-10-21 00:52:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6324 [2026-10-21 00:52:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5016 [2026-10-21 00:52:01][INFO][training.pretrain_engine] [Ph2|Step 756] loss=5.3938 | ppl=220.04 | aux=0.02509 | lr=3.00e-04 | gnorm=0.660 | tok/s=29,284 | MFU=7.0% | step=143229ms | tokens=0.800B | mem=4.2/32.7GB | cost=$397.25/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 15:33 (134.7h) | Phase3: SKIPPED | SFT: Oct 26 20:18 (139.4h) | DPO (Final): Oct 26 20:52 (140.0h) [2026-10-21 00:52:01][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $397.25 of $400.00 [2026-10-21 00:52:01][INFO][training.pretrain_engine] Starting global step 757 (accumulating 32 micro-batches)... [2026-10-21 00:52:01][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $397.25 of $400.00 [2026-10-21 00:52:01][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $397.25 of $400.00 [2026-10-21 00:52:01][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 00:52:01][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $397.25 of $400.00 [2026-10-21 00:52:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7434 [2026-10-21 00:52:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6193 [2026-10-21 00:52:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7575 [2026-10-21 00:52:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0362 [2026-10-21 00:52:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2805 [2026-10-21 00:52:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3912 [2026-10-21 00:52:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2101 [2026-10-21 00:52:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5096 [2026-10-21 00:52:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0050 [2026-10-21 00:52:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5586 [2026-10-21 00:52:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7375 [2026-10-21 00:52:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3562 [2026-10-21 00:53:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3122 [2026-10-21 00:53:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5701 [2026-10-21 00:53:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1487 [2026-10-21 00:53:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0716 [2026-10-21 00:53:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2388 [2026-10-21 00:53:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0146 [2026-10-21 00:53:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6398 [2026-10-21 00:53:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1072 [2026-10-21 00:53:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5222 [2026-10-21 00:53:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4184 [2026-10-21 00:53:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4827 [2026-10-21 00:53:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4688 [2026-10-21 00:54:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3882 [2026-10-21 00:54:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0281 [2026-10-21 00:54:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6761 [2026-10-21 00:54:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2691 [2026-10-21 00:54:19][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,602 | MFU: 1.8% [2026-10-21 00:54:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6359 [2026-10-21 00:54:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0687 [2026-10-21 00:54:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4012 [2026-10-21 00:54:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8837 [2026-10-21 00:54:24][INFO][training.pretrain_engine] [Ph2|Step 757] loss=5.4853 | ppl=241.11 | aux=0.02508 | lr=3.00e-04 | gnorm=0.586 | tok/s=29,284 | MFU=7.0% | step=143229ms | tokens=0.801B | mem=4.2/32.7GB | cost=$397.78/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 15:35 (134.7h) | Phase3: SKIPPED | SFT: Oct 26 20:20 (139.4h) | DPO (Final): Oct 26 20:54 (140.0h) [2026-10-21 00:54:24][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $397.78 of $400.00 [2026-10-21 00:54:24][INFO][training.pretrain_engine] Starting global step 758 (accumulating 32 micro-batches)... [2026-10-21 00:54:24][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $397.78 of $400.00 [2026-10-21 00:54:24][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $397.78 of $400.00 [2026-10-21 00:54:24][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $397.78 of $400.00 [2026-10-21 00:54:24][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 00:54:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7370 [2026-10-21 00:54:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4025 [2026-10-21 00:54:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0361 [2026-10-21 00:54:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0753 [2026-10-21 00:55:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4869 [2026-10-21 00:55:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2754 [2026-10-21 00:55:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8625 [2026-10-21 00:55:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4957 [2026-10-21 00:55:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3101 [2026-10-21 00:55:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5336 [2026-10-21 00:55:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7419 [2026-10-21 00:55:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3659 [2026-10-21 00:55:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3372 [2026-10-21 00:55:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0671 [2026-10-21 00:55:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3204 [2026-10-21 00:55:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4913 [2026-10-21 00:55:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5176 [2026-10-21 00:55:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4579 [2026-10-21 00:55:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6095 [2026-10-21 00:55:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6415 [2026-10-21 00:56:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4783 [2026-10-21 00:56:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8462 [2026-10-21 00:56:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1245 [2026-10-21 00:56:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2819 [2026-10-21 00:56:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7416 [2026-10-21 00:56:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7316 [2026-10-21 00:56:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6578 [2026-10-21 00:56:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5607 [2026-10-21 00:56:42][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,605 | MFU: 1.8% [2026-10-21 00:56:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0231 [2026-10-21 00:56:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5373 [2026-10-21 00:56:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7575 [2026-10-21 00:56:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8215 [2026-10-21 00:56:48][INFO][training.pretrain_engine] [Ph2|Step 758] loss=5.4703 | ppl=237.52 | aux=0.02507 | lr=3.00e-04 | gnorm=0.625 | tok/s=29,284 | MFU=7.0% | step=143229ms | tokens=0.802B | mem=4.2/32.7GB | cost=$398.30/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 15:37 (134.7h) | Phase3: SKIPPED | SFT: Oct 26 20:21 (139.4h) | DPO (Final): Oct 26 20:56 (140.0h) [2026-10-21 00:56:48][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $398.30 of $400.00 [2026-10-21 00:56:48][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $398.30 of $400.00 [2026-10-21 00:56:48][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $398.30 of $400.00 [2026-10-21 00:56:51][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step758 | Tokens: 0.802B | Spent: $398.31 | Remaining: $1.69 [2026-10-21 00:56:51][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step758 [2026-10-21 00:56:51][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $398.31 of $400.00 [2026-10-21 00:56:51][INFO][training.pretrain_engine] Starting global step 759 (accumulating 32 micro-batches)... [2026-10-21 00:56:51][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 00:56:51][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 00:56:51][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 00:56:51][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 00:56:51][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step758 to Phase-Technologies/forge-500m-pretrain (path: phase2_step758)... [2026-10-21 00:56:51][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 00:57:02][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 00:57:02][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step758 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 00:57:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6156 [2026-10-21 00:57:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3975 [2026-10-21 00:57:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5585 [2026-10-21 00:57:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6631 [2026-10-21 00:57:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1085 [2026-10-21 00:57:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2093 [2026-10-21 00:57:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8619 [2026-10-21 00:57:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9482 [2026-10-21 00:57:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3316 [2026-10-21 00:57:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0079 [2026-10-21 00:57:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4379 [2026-10-21 00:57:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7694 [2026-10-21 00:57:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2771 [2026-10-21 00:57:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6060 [2026-10-21 00:57:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5745 [2026-10-21 00:58:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3478 [2026-10-21 00:58:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9800 [2026-10-21 00:58:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8811 [2026-10-21 00:58:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7541 [2026-10-21 00:58:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1680 [2026-10-21 00:58:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5141 [2026-10-21 00:58:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5310 [2026-10-21 00:58:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5231 [2026-10-21 00:58:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4195 [2026-10-21 00:58:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6125 [2026-10-21 00:58:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4754 [2026-10-21 00:58:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4105 [2026-10-21 00:58:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6155 [2026-10-21 00:59:08][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,609 | MFU: 1.8% [2026-10-21 00:59:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8581 [2026-10-21 00:59:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4235 [2026-10-21 00:59:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0323 [2026-10-21 00:59:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5429 [2026-10-21 00:59:13][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $398.83 of $400.00 [2026-10-21 00:59:13][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $398.83 of $400.00 [2026-10-21 00:59:13][INFO][training.pretrain_engine] [Ph2|Step 759] loss=5.3603 | ppl=212.79 | aux=0.02508 | lr=3.00e-04 | gnorm=0.586 | tok/s=29,286 | MFU=7.0% | step=143218ms | tokens=0.803B | mem=4.2/32.7GB | cost=$398.83/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 15:38 (134.7h) | Phase3: SKIPPED | SFT: Oct 26 20:23 (139.4h) | DPO (Final): Oct 26 20:57 (140.0h) [2026-10-21 00:59:13][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $398.83 of $400.00 [2026-10-21 00:59:13][INFO][training.pretrain_engine] Starting global step 760 (accumulating 32 micro-batches)... [2026-10-21 00:59:13][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $398.83 of $400.00 [2026-10-21 00:59:13][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 00:59:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5518 [2026-10-21 00:59:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1514 [2026-10-21 00:59:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5007 [2026-10-21 00:59:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3800 [2026-10-21 00:59:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4680 [2026-10-21 00:59:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2713 [2026-10-21 00:59:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4161 [2026-10-21 00:59:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3267 [2026-10-21 01:00:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5793 [2026-10-21 01:00:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3050 [2026-10-21 01:00:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7214 [2026-10-21 01:00:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6609 [2026-10-21 01:00:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4949 [2026-10-21 01:00:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5247 [2026-10-21 01:00:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4716 [2026-10-21 01:00:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1589 [2026-10-21 01:00:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1299 [2026-10-21 01:00:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2405 [2026-10-21 01:00:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0933 [2026-10-21 01:00:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2644 [2026-10-21 01:01:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4341 [2026-10-21 01:01:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8416 [2026-10-21 01:01:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4868 [2026-10-21 01:01:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7004 [2026-10-21 01:01:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6522 [2026-10-21 01:01:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2309 [2026-10-21 01:01:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0339 [2026-10-21 01:01:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2148 [2026-10-21 01:01:31][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,592 | MFU: 1.8% [2026-10-21 01:01:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0601 [2026-10-21 01:01:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8525 [2026-10-21 01:01:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6726 [2026-10-21 01:01:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7458 [2026-10-21 01:01:36][INFO][training.pretrain_engine] [Ph2|Step 760] loss=5.4029 | ppl=222.06 | aux=0.02508 | lr=3.00e-04 | gnorm=0.590 | tok/s=29,286 | MFU=7.0% | step=143219ms | tokens=0.804B | mem=4.2/32.7GB | cost=$399.36/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 15:40 (134.6h) | Phase3: SKIPPED | SFT: Oct 26 20:24 (139.4h) | DPO (Final): Oct 26 20:59 (140.0h) [2026-10-21 01:01:36][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $399.36 of $400.00 [2026-10-21 01:01:36][INFO][training.pretrain_engine] Starting global step 761 (accumulating 32 micro-batches)... [2026-10-21 01:01:36][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 01:01:36][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $399.36 of $400.00 [2026-10-21 01:01:36][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $399.36 of $400.00 [2026-10-21 01:01:36][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $399.36 of $400.00 [2026-10-21 01:01:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6419 [2026-10-21 01:01:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5528 [2026-10-21 01:01:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0340 [2026-10-21 01:01:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5404 [2026-10-21 01:02:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5146 [2026-10-21 01:02:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1904 [2026-10-21 01:02:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3334 [2026-10-21 01:02:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0104 [2026-10-21 01:02:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8549 [2026-10-21 01:02:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5505 [2026-10-21 01:02:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9959 [2026-10-21 01:02:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3515 [2026-10-21 01:02:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0115 [2026-10-21 01:02:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6883 [2026-10-21 01:02:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6393 [2026-10-21 01:02:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5643 [2026-10-21 01:03:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4686 [2026-10-21 01:03:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4667 [2026-10-21 01:03:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2955 [2026-10-21 01:03:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7512 [2026-10-21 01:03:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5041 [2026-10-21 01:03:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7912 [2026-10-21 01:03:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9391 [2026-10-21 01:03:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5667 [2026-10-21 01:03:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2170 [2026-10-21 01:03:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0790 [2026-10-21 01:03:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0266 [2026-10-21 01:03:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1775 [2026-10-21 01:03:54][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,621 | MFU: 1.8% [2026-10-21 01:03:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6767 [2026-10-21 01:03:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2499 [2026-10-21 01:03:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4889 [2026-10-21 01:03:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4746 [2026-10-21 01:03:59][INFO][training.pretrain_engine] [Ph2|Step 761] loss=5.3951 | ppl=220.33 | aux=0.02506 | lr=3.00e-04 | gnorm=0.582 | tok/s=29,287 | MFU=7.0% | step=143212ms | tokens=0.805B | mem=4.2/32.7GB | cost=$399.88/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 15:41 (134.6h) | Phase3: SKIPPED | SFT: Oct 26 20:26 (139.4h) | DPO (Final): Oct 26 21:00 (139.9h) [2026-10-21 01:03:59][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $399.88 of $400.00 [2026-10-21 01:03:59][INFO][training.pretrain_engine] Starting global step 762 (accumulating 32 micro-batches)... [2026-10-21 01:03:59][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $399.88 of $400.00 [2026-10-21 01:03:59][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $399.88 of $400.00 [2026-10-21 01:03:59][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $399.88 of $400.00 [2026-10-21 01:03:59][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 01:04:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7345 [2026-10-21 01:04:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4468 [2026-10-21 01:04:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6373 [2026-10-21 01:04:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6330 [2026-10-21 01:04:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5096 [2026-10-21 01:04:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4243 [2026-10-21 01:04:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2731 [2026-10-21 01:04:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6439 [2026-10-21 01:04:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6048 [2026-10-21 01:04:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5220 [2026-10-21 01:04:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1781 [2026-10-21 01:04:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3931 [2026-10-21 01:05:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2020 [2026-10-21 01:05:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3918 [2026-10-21 01:05:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4641 [2026-10-21 01:05:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5513 [2026-10-21 01:05:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3935 [2026-10-21 01:05:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4646 [2026-10-21 01:05:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7462 [2026-10-21 01:05:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7547 [2026-10-21 01:05:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8698 [2026-10-21 01:05:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7293 [2026-10-21 01:05:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6902 [2026-10-21 01:05:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9824 [2026-10-21 01:06:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3936 [2026-10-21 01:06:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7580 [2026-10-21 01:06:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7260 [2026-10-21 01:06:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2639 [2026-10-21 01:06:16][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,611 | MFU: 1.8% [2026-10-21 01:06:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6491 [2026-10-21 01:06:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0608 [2026-10-21 01:06:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8953 [2026-10-21 01:06:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5536 [2026-10-21 01:06:21][INFO][training.pretrain_engine] [Ph2|Step 762] loss=5.3985 | ppl=221.07 | aux=0.02508 | lr=3.00e-04 | gnorm=0.629 | tok/s=29,289 | MFU=7.0% | step=143206ms | tokens=0.806B | mem=4.2/32.7GB | cost=$400.40/$400 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 15:43 (134.6h) | Phase3: SKIPPED | SFT: Oct 26 20:27 (139.4h) | DPO (Final): Oct 26 21:01 (139.9h) [2026-10-21 01:06:21][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $400.40 of $400.00 [2026-10-21 01:06:21][ERROR][training.pretrain_engine] ❌ Budget exceeded: $400.40 > $400.00. Stopping! [2026-10-21 01:06:21][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $400.40 of $400.00 [2026-10-21 01:06:21][ERROR][training.pretrain_engine] ❌ Budget exceeded: $400.40 > $400.00. Stopping! [2026-10-21 01:06:21][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $400.40 of $400.00 [2026-10-21 01:06:21][ERROR][training.pretrain_engine] ❌ Budget exceeded: $400.40 > $400.00. Stopping! [2026-10-21 01:06:21][WARNING][training.pretrain_engine] ⚠️ Budget at 95%: $400.40 of $400.00 [2026-10-21 01:06:21][ERROR][training.pretrain_engine] ❌ Budget exceeded: $400.40 > $400.00. Stopping! W1021 01:06:23.477000 2022 torch/distributed/elastic/multiprocessing/api.py:1028] Sending process 2104 closing signal SIGTERM W1021 01:06:23.478000 2022 torch/distributed/elastic/multiprocessing/api.py:1028] Sending process 2106 closing signal SIGTERM W1021 01:06:23.479000 2022 torch/distributed/elastic/multiprocessing/api.py:1028] Sending process 2107 closing signal SIGTERM E1021 01:06:23.858000 2022 torch/distributed/elastic/multiprocessing/api.py:1002] failed (exitcode: 1) local_rank: 1 (pid: 2105) of binary: /usr/bin/python3.12 Traceback (most recent call last): File "/usr/local/bin/torchrun", line 7, in sys.exit(main()) ^^^^^^ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 380, in wrapper return f(*args, **kwargs) ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 1138, in main run(args) File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 1129, in run elastic_launch( File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 197, in __call__ return launch_agent( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 386, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError: ============================================================ run_pretrain.py FAILED ------------------------------------------------------------ Failures: [1]: time : 2026-10-21_01:06:23 host : fc13a1f48d0d rank : 0 (local_rank: 0) exitcode : -15 (pid: 2104) (SIGTERM) error_file: traceback : Signal 15 (SIGTERM) received by PID 2104 [2]: time : 2026-10-21_01:06:23 host : fc13a1f48d0d rank : 2 (local_rank: 2) exitcode : 1 (pid: 2106) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [3]: time : 2026-10-21_01:06:23 host : fc13a1f48d0d rank : 3 (local_rank: 3) exitcode : 1 (pid: 2107) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ------------------------------------------------------------ Root Cause (first observed failure): [0]: time : 2026-10-21_01:06:23 host : fc13a1f48d0d rank : 1 (local_rank: 1) exitcode : 1 (pid: 2105) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ============================================================ 🔄 [AUTO-RESUME] Found existing checkpoint: /workspace/checkpoints/forge_500m_pretrain/phase2_step758 (resuming automatically!) [INFO] Logging to /workspace/checkpoints/forge_500m_pretrain/logs/pretrain_20261021_064838.log ======================================================================== FORGE-500M PRETRAINING ======================================================================== Model config : ./configs/forge_500m.json Data dir : /workspace/data/tokenized Output dir : /workspace/checkpoints/forge_500m_pretrain GPUs : 4 Phase 1 tokens : 0B Phase 2 tokens : 15B Phase 3 tokens : 0B Total tokens : 15B Micro batch/GPU : Phase1=4 | Phase2=4 | Phase3=4 Global batch : 1048576 tokens/step (Phase 2) Resume from : /workspace/checkpoints/forge_500m_pretrain/phase2_step758 Budget cap : $400 ======================================================================== W1021 06:48:39.743000 2746 torch/distributed/run.py:982] W1021 06:48:39.743000 2746 torch/distributed/run.py:982] ***************************************** W1021 06:48:39.743000 2746 torch/distributed/run.py:982] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. W1021 06:48:39.743000 2746 torch/distributed/run.py:982] ***************************************** Traceback (most recent call last): File "/usr/local/bin/torchrun", line 7, in sys.exit(main()) ^^^^^^ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 380, in wrapper return f(*args, **kwargs) ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 1138, in main run(args) File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 1129, in run elastic_launch( File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 197, in __call__ return launch_agent( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 377, in launch_agent result = agent.run() ^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/metrics/api.py", line 134, in wrapper result = f(*args, **kwargs) ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/agent/server/api.py", line 745, in run result = self._invoke_run(role) ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/agent/server/api.py", line 916, in _invoke_run self._initialize_workers(self._worker_group) File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/metrics/api.py", line 134, in wrapper result = f(*args, **kwargs) ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/agent/server/api.py", line 713, in _initialize_workers self._rendezvous(worker_group) File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/metrics/api.py", line 134, in wrapper result = f(*args, **kwargs) ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/agent/server/api.py", line 528, in _rendezvous rdzv_info = spec.rdzv_handler.next_rendezvous() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/rendezvous/static_tcp_rendezvous.py", line 67, in next_rendezvous self._store = TCPStore( # type: ignore[call-arg] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 29500, useIpv6: false, code: -98, name: EADDRINUSE, message: address already in use 🔄 [AUTO-RESUME] Found existing checkpoint: /workspace/checkpoints/forge_500m_pretrain/phase2_step758 (resuming automatically!) [INFO] Logging to /workspace/checkpoints/forge_500m_pretrain/logs/pretrain_20261021_064915.log ======================================================================== FORGE-500M PRETRAINING ======================================================================== Model config : ./configs/forge_500m.json Data dir : /workspace/data/tokenized Output dir : /workspace/checkpoints/forge_500m_pretrain GPUs : 4 Phase 1 tokens : 0B Phase 2 tokens : 15B Phase 3 tokens : 0B Total tokens : 15B Micro batch/GPU : Phase1=4 | Phase2=4 | Phase3=4 Global batch : 1048576 tokens/step (Phase 2) Resume from : /workspace/checkpoints/forge_500m_pretrain/phase2_step758 Budget cap : $400 ======================================================================== W1021 06:49:16.800000 2977 torch/distributed/run.py:982] W1021 06:49:16.800000 2977 torch/distributed/run.py:982] ***************************************** W1021 06:49:16.800000 2977 torch/distributed/run.py:982] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. W1021 06:49:16.800000 2977 torch/distributed/run.py:982] ***************************************** [2026-10-21 06:49:19][INFO][training.gpu_optimizer] Distributed initialized: rank=1/4, local=1 [2026-10-21 06:49:19][INFO][training.gpu_optimizer] Distributed initialized: rank=0/4, local=0 [2026-10-21 06:49:19][INFO][training.gpu_optimizer] Distributed initialized: rank=2/4, local=2 [2026-10-21 06:49:19][INFO][__main__] ====================================================================== [2026-10-21 06:49:19][INFO][training.gpu_optimizer] Distributed initialized: rank=3/4, local=3 [2026-10-21 06:49:19][INFO][__main__] FORGE-500M PRETRAINING [2026-10-21 06:49:19][INFO][__main__] Rank: 0/4 [2026-10-21 06:49:19][INFO][__main__] Device: cuda:0 [2026-10-21 06:49:19][INFO][__main__] Data: /workspace/data/tokenized [2026-10-21 06:49:19][INFO][__main__] Output: /workspace/checkpoints/forge_500m_pretrain [2026-10-21 06:49:19][INFO][__main__] ====================================================================== [2026-10-21 06:49:19][INFO][__main__] Batch tokens — Phase1: 1,048,576 | Phase2: 1,048,576 | Phase3: 1,048,576 [2026-10-21 06:49:19][INFO][__main__] Loading CRAYON tokenizer (profile=standard)... [2026-10-21 06:49:19][INFO][__main__] Batch tokens — Phase1: 1,048,576 | Phase2: 1,048,576 | Phase3: 1,048,576 [2026-10-21 06:49:19][INFO][__main__] Batch tokens — Phase1: 1,048,576 | Phase2: 1,048,576 | Phase3: 1,048,576 [2026-10-21 06:49:19][INFO][__main__] Loading CRAYON tokenizer (profile=standard)... [2026-10-21 06:49:19][INFO][__main__] Loading CRAYON tokenizer (profile=standard)... [2026-10-21 06:49:19][INFO][__main__] Batch tokens — Phase1: 1,048,576 | Phase2: 1,048,576 | Phase3: 1,048,576 [2026-10-21 06:49:19][INFO][__main__] Loading CRAYON tokenizer (profile=standard)... 🚀 TURBO ENGINE LOADED: Intel(R) Xeon(R) 6787P [Turbo/v5.7.15/64K-SA+AVX2/OMP 64Ksets×2ways intcache=0] [2026-10-21 06:49:19][INFO][crayon.vocab] Turbo engine loaded successfully ✅ CPU BACKEND: Compiled C++ Extension (fallback) Hardware: Intel(R) Xeon(R) 6787P [Standard] ✅ DEVICE READY: CPU (Compiled C++) Hardware: Intel(R) Xeon(R) 6787P [Standard] [2026-10-21 06:49:19][INFO][crayon.vocab] 🔵 CPU Engine Active: Intel(R) Xeon(R) 6787P [Standard] 🔧 INITIALIZING DEVICE: CPU 🚀 TURBO ENGINE LOADED: Intel(R) Xeon(R) 6787P [Turbo/v5.7.15/64K-SA+AVX2/OMP 64Ksets×2ways intcache=0] [2026-10-21 06:49:19][INFO][crayon.vocab] Turbo engine loaded successfully ✅ CPU BACKEND: Compiled C++ Extension (fallback) Hardware: Intel(R) Xeon(R) 6787P [Standard] ✅ DEVICE READY: CPU (Compiled C++) Hardware: Intel(R) Xeon(R) 6787P [Standard] 🚀 TURBO ENGINE LOADED: Intel(R) Xeon(R) 6787P [Turbo/v5.7.15/64K-SA+AVX2/OMP 64Ksets×2ways intcache=0] [2026-10-21 06:49:19][INFO][crayon.vocab] 🔵 CPU Engine Active: Intel(R) Xeon(R) 6787P [Standard] 🔧 INITIALIZING DEVICE: CPU [2026-10-21 06:49:19][INFO][crayon.vocab] Turbo engine loaded successfully ✅ CPU BACKEND: Compiled C++ Extension (fallback) Hardware: Intel(R) Xeon(R) 6787P [Standard] ✅ DEVICE READY: CPU (Compiled C++) Hardware: Intel(R) Xeon(R) 6787P [Standard] [2026-10-21 06:49:19][INFO][crayon.vocab] 🔵 CPU Engine Active: Intel(R) Xeon(R) 6787P [Standard] 🔧 INITIALIZING DEVICE: CPU 🚀 TURBO ENGINE LOADED: Intel(R) Xeon(R) 6787P [Turbo/v5.7.15/64K-SA+AVX2/OMP 64Ksets×2ways intcache=0] [2026-10-21 06:49:19][INFO][crayon.vocab] Turbo engine loaded successfully ✅ CPU BACKEND: Compiled C++ Extension (fallback) Hardware: Intel(R) Xeon(R) 6787P [Standard] ✅ DEVICE READY: CPU (Compiled C++) Hardware: Intel(R) Xeon(R) 6787P [Standard] [2026-10-21 06:49:19][INFO][crayon.vocab] 🔵 CPU Engine Active: Intel(R) Xeon(R) 6787P [Standard] 🔧 INITIALIZING DEVICE: CPU [2026-10-21 06:49:19][INFO][tokenizer.crayon_wrapper] ForgeTokenizer initialized: profile=standard, vocab_size=206464 (crayon=206373 + specials=13 + padding=78), n_workers=172, throughput ~24M tok/s (CPU) [2026-10-21 06:49:19][INFO][__main__] Building FORGE-500M model... [2026-10-21 06:49:19][INFO][tokenizer.crayon_wrapper] ForgeTokenizer initialized: profile=standard, vocab_size=206464 (crayon=206373 + specials=13 + padding=78), n_workers=172, throughput ~24M tok/s (CPU) [2026-10-21 06:49:19][INFO][__main__] Tokenizer: ForgeTokenizer(profile='standard', vocab_size=206,464, n_workers=172) [2026-10-21 06:49:19][INFO][tokenizer.crayon_wrapper] Tokenizer config saved to /workspace/checkpoints/forge_500m_pretrain/tokenizer [2026-10-21 06:49:19][INFO][__main__] Building FORGE-500M model... [2026-10-21 06:49:19][INFO][tokenizer.crayon_wrapper] ForgeTokenizer initialized: profile=standard, vocab_size=206464 (crayon=206373 + specials=13 + padding=78), n_workers=172, throughput ~24M tok/s (CPU) [2026-10-21 06:49:19][INFO][__main__] Building FORGE-500M model... [2026-10-21 06:49:19][INFO][tokenizer.crayon_wrapper] ForgeTokenizer initialized: profile=standard, vocab_size=206464 (crayon=206373 + specials=13 + padding=78), n_workers=172, throughput ~24M tok/s (CPU) [2026-10-21 06:49:19][INFO][__main__] Building FORGE-500M model... /usr/local/lib/python3.12/dist-packages/torch/jit/_script.py:1491: FutureWarning: `torch.jit.script` is deprecated. Please switch to `torch.compile` or `torch.export`. warnings.warn( /usr/local/lib/python3.12/dist-packages/torch/jit/_script.py:1491: FutureWarning: `torch.jit.script` is deprecated. Please switch to `torch.compile` or `torch.export`. warnings.warn( /usr/local/lib/python3.12/dist-packages/torch/jit/_script.py:1491: FutureWarning: `torch.jit.script` is deprecated. Please switch to `torch.compile` or `torch.export`. warnings.warn( /usr/local/lib/python3.12/dist-packages/torch/jit/_script.py:1491: FutureWarning: `torch.jit.script` is deprecated. Please switch to `torch.compile` or `torch.export`. warnings.warn( [2026-10-21 06:49:28][INFO][model.forge_model] ForgeModel: 0.500B total parameters [2026-10-21 06:49:28][INFO][model.forge_model] Embedding: 211.4M [2026-10-21 06:49:28][INFO][model.forge_model] N layers: 18 (3 MHA + 15 ARG, 9 HSE + 9 Dense) [2026-10-21 06:49:28][INFO][model.forge_model] ForgeModel: 0.500B total parameters [2026-10-21 06:49:28][INFO][model.forge_model] Embedding: 211.4M [2026-10-21 06:49:28][INFO][model.forge_model] N layers: 18 (3 MHA + 15 ARG, 9 HSE + 9 Dense) [2026-10-21 06:49:28][INFO][model.forge_model] ForgeModel: 0.500B total parameters [2026-10-21 06:49:28][INFO][model.forge_model] Embedding: 211.4M [2026-10-21 06:49:28][INFO][model.forge_model] N layers: 18 (3 MHA + 15 ARG, 9 HSE + 9 Dense) [2026-10-21 06:49:28][INFO][model.forge_model] ForgeModel: 0.500B total parameters [2026-10-21 06:49:28][INFO][model.forge_model] Embedding: 211.4M [2026-10-21 06:49:28][INFO][model.forge_model] N layers: 18 (3 MHA + 15 ARG, 9 HSE + 9 Dense) [2026-10-21 06:49:28][INFO][model.forge_model] Gradient checkpointing enabled [2026-10-21 06:49:28][INFO][__main__] Gradient checkpointing enabled [2026-10-21 06:49:28][INFO][__main__] Resuming from checkpoint: /workspace/checkpoints/forge_500m_pretrain/phase2_step758 [2026-10-21 06:49:28][INFO][__main__] Model: 0.500B total params, 0.500B trainable [2026-10-21 06:49:28][WARNING][__main__] ⚠ Parameter count 0.500B is outside the expected [0.8B, 1.1B] range. Check model_config! [2026-10-21 06:49:28][INFO][model.forge_model] Gradient checkpointing enabled [2026-10-21 06:49:28][INFO][__main__] Gradient checkpointing enabled [2026-10-21 06:49:28][INFO][__main__] Resuming from checkpoint: /workspace/checkpoints/forge_500m_pretrain/phase2_step758 [2026-10-21 06:49:29][INFO][model.forge_model] Gradient checkpointing enabled [2026-10-21 06:49:29][INFO][__main__] Gradient checkpointing enabled [2026-10-21 06:49:29][INFO][__main__] Resuming from checkpoint: /workspace/checkpoints/forge_500m_pretrain/phase2_step758 [2026-10-21 06:49:29][INFO][model.forge_model] Gradient checkpointing enabled [2026-10-21 06:49:29][INFO][__main__] Gradient checkpointing enabled [2026-10-21 06:49:29][INFO][__main__] Resuming from checkpoint: /workspace/checkpoints/forge_500m_pretrain/phase2_step758 [2026-10-21 06:49:29][INFO][__main__] Model weights loaded from checkpoint [2026-10-21 06:49:29][INFO][__main__] Resuming at 0.80B tokens already processed [2026-10-21 06:49:29][INFO][data.dataset] Raised open file limit (RLIMIT_NOFILE): 1024 → 65536 [2026-10-21 06:49:29][INFO][__main__] Phase 1 disabled (phase1_tokens=0) — starting directly on Phase 2. [2026-10-21 06:49:29][INFO][__main__] Building Phase-2 dataloader (seq=2048, full domain mix)... [2026-10-21 06:49:29][INFO][__main__] Model weights loaded from checkpoint [2026-10-21 06:49:29][INFO][__main__] Resuming at 0.80B tokens already processed [2026-10-21 06:49:29][INFO][__main__] Phase 1 disabled (phase1_tokens=0) — starting directly on Phase 2. [2026-10-21 06:49:29][INFO][__main__] Building Phase-2 dataloader (seq=2048, full domain mix)... [2026-10-21 06:49:29][INFO][__main__] Model weights loaded from checkpoint [2026-10-21 06:49:29][INFO][__main__] Resuming at 0.80B tokens already processed [2026-10-21 06:49:29][INFO][data.dataset] Raised open file limit (RLIMIT_NOFILE): 1024 → 65536 [2026-10-21 06:49:29][INFO][__main__] Phase 1 disabled (phase1_tokens=0) — starting directly on Phase 2. [2026-10-21 06:49:29][INFO][__main__] Building Phase-2 dataloader (seq=2048, full domain mix)... [2026-10-21 06:49:29][INFO][__main__] Model weights loaded from checkpoint [2026-10-21 06:49:29][INFO][data.dataset] PackedTokenDataset (train): 7,309,985 sequences × 2048 tokens = 14.97B tokens total [2026-10-21 06:49:29][INFO][__main__] Loaded 'fineweb_edu': 7,309,985 sequences [2026-10-21 06:49:29][INFO][__main__] Resuming at 0.80B tokens already processed [2026-10-21 06:49:29][INFO][data.dataset] Raised open file limit (RLIMIT_NOFILE): 1024 → 65536 [2026-10-21 06:49:29][INFO][__main__] Phase 1 disabled (phase1_tokens=0) — starting directly on Phase 2. [2026-10-21 06:49:29][INFO][__main__] Building Phase-2 dataloader (seq=2048, full domain mix)... [2026-10-21 06:49:29][INFO][data.dataset] PackedTokenDataset (train): 7,309,985 sequences × 2048 tokens = 14.97B tokens total [2026-10-21 06:49:29][INFO][__main__] Loaded 'fineweb_edu': 7,309,985 sequences [2026-10-21 06:49:29][INFO][data.dataset] PackedTokenDataset (train): 3,719,165 sequences × 2048 tokens = 7.62B tokens total [2026-10-21 06:49:29][INFO][__main__] Loaded 'thestack': 3,719,165 sequences [2026-10-21 06:49:29][INFO][data.dataset] PackedTokenDataset (train): 3,719,165 sequences × 2048 tokens = 7.62B tokens total [2026-10-21 06:49:29][INFO][__main__] Loaded 'thestack': 3,719,165 sequences [2026-10-21 06:49:29][INFO][data.dataset] PackedTokenDataset (train): 2,088,970 sequences × 2048 tokens = 4.28B tokens total [2026-10-21 06:49:29][INFO][__main__] Loaded 'wikipedia': 2,088,970 sequences [2026-10-21 06:49:29][INFO][data.dataset] PackedTokenDataset (train): 2,088,970 sequences × 2048 tokens = 4.28B tokens total [2026-10-21 06:49:29][INFO][__main__] Loaded 'wikipedia': 2,088,970 sequences [2026-10-21 06:49:29][INFO][data.dataset] PackedTokenDataset (train): 1,935,817 sequences × 2048 tokens = 3.96B tokens total [2026-10-21 06:49:29][INFO][__main__] Loaded 'openwebmath': 1,935,817 sequences [2026-10-21 06:49:29][INFO][data.dataset] PackedTokenDataset (train): 7,309,985 sequences × 2048 tokens = 14.97B tokens total [2026-10-21 06:49:29][INFO][__main__] Loaded 'fineweb_edu': 7,309,985 sequences [2026-10-21 06:49:29][INFO][data.dataset] PackedTokenDataset (train): 7,309,985 sequences × 2048 tokens = 14.97B tokens total [2026-10-21 06:49:29][INFO][__main__] Loaded 'fineweb_edu': 7,309,985 sequences [2026-10-21 06:49:29][INFO][data.dataset] PackedTokenDataset (train): 1,935,817 sequences × 2048 tokens = 3.96B tokens total [2026-10-21 06:49:29][INFO][__main__] Loaded 'openwebmath': 1,935,817 sequences [2026-10-21 06:49:29][INFO][data.dataset] PackedTokenDataset (train): 1,350,980 sequences × 2048 tokens = 2.77B tokens total [2026-10-21 06:49:29][INFO][__main__] Loaded 'books': 1,350,980 sequences [2026-10-21 06:49:29][INFO][data.dataset] PackedTokenDataset (train): 1,350,980 sequences × 2048 tokens = 2.77B tokens total [2026-10-21 06:49:29][INFO][__main__] Loaded 'books': 1,350,980 sequences [2026-10-21 06:49:29][INFO][data.dataset] PackedTokenDataset (train): 3,719,165 sequences × 2048 tokens = 7.62B tokens total [2026-10-21 06:49:29][INFO][__main__] Loaded 'thestack': 3,719,165 sequences [2026-10-21 06:49:29][INFO][data.dataset] PackedTokenDataset (train): 3,719,165 sequences × 2048 tokens = 7.62B tokens total [2026-10-21 06:49:29][INFO][__main__] Loaded 'thestack': 3,719,165 sequences [2026-10-21 06:49:29][INFO][data.dataset] PackedTokenDataset (train): 3,202,429 sequences × 2048 tokens = 6.56B tokens total [2026-10-21 06:49:29][INFO][__main__] Loaded 'arxiv': 3,202,429 sequences [2026-10-21 06:49:29][INFO][data.dataset] PackedTokenDataset (train): 3,202,429 sequences × 2048 tokens = 6.56B tokens total [2026-10-21 06:49:29][INFO][__main__] Loaded 'arxiv': 3,202,429 sequences [2026-10-21 06:49:30][INFO][data.dataset] PackedTokenDataset (train): 2,088,970 sequences × 2048 tokens = 4.28B tokens total [2026-10-21 06:49:30][INFO][__main__] Loaded 'wikipedia': 2,088,970 sequences [2026-10-21 06:49:30][INFO][data.dataset] PackedTokenDataset (train): 2,088,970 sequences × 2048 tokens = 4.28B tokens total [2026-10-21 06:49:30][INFO][__main__] Loaded 'wikipedia': 2,088,970 sequences [2026-10-21 06:49:30][INFO][data.dataset] PackedTokenDataset (train): 2,418,666 sequences × 2048 tokens = 4.95B tokens total [2026-10-21 06:49:30][INFO][__main__] Loaded 'dolma': 2,418,666 sequences [2026-10-21 06:49:30][INFO][data.dataset] PackedTokenDataset (train): 1,935,817 sequences × 2048 tokens = 3.96B tokens total [2026-10-21 06:49:30][INFO][__main__] Loaded 'openwebmath': 1,935,817 sequences [2026-10-21 06:49:30][INFO][data.dataset] PackedTokenDataset (train): 2,418,666 sequences × 2048 tokens = 4.95B tokens total [2026-10-21 06:49:30][INFO][__main__] Loaded 'dolma': 2,418,666 sequences [2026-10-21 06:49:30][INFO][data.dataset] PackedTokenDataset (train): 1,935,817 sequences × 2048 tokens = 3.96B tokens total [2026-10-21 06:49:30][INFO][__main__] Loaded 'openwebmath': 1,935,817 sequences [2026-10-21 06:49:30][INFO][data.dataset] PackedTokenDataset (train): 1,178,010 sequences × 2048 tokens = 2.41B tokens total [2026-10-21 06:49:30][INFO][__main__] Loaded 'stackexchange': 1,178,010 sequences [2026-10-21 06:49:30][INFO][data.dataset] PackedTokenDataset (train): 1,178,010 sequences × 2048 tokens = 2.41B tokens total [2026-10-21 06:49:30][INFO][__main__] Loaded 'stackexchange': 1,178,010 sequences [2026-10-21 06:49:30][INFO][data.dataset] PackedTokenDataset (train): 1,350,980 sequences × 2048 tokens = 2.77B tokens total [2026-10-21 06:49:30][INFO][__main__] Loaded 'books': 1,350,980 sequences [2026-10-21 06:49:30][INFO][data.dataset] PackedTokenDataset (train): 1,350,980 sequences × 2048 tokens = 2.77B tokens total [2026-10-21 06:49:30][INFO][__main__] Loaded 'books': 1,350,980 sequences [2026-10-21 06:49:30][INFO][data.dataset] PackedTokenDataset (train): 520,865 sequences × 2048 tokens = 1.07B tokens total [2026-10-21 06:49:30][INFO][__main__] Loaded 'redpajama_cc': 520,865 sequences [2026-10-21 06:49:30][INFO][data.dataset] PackedTokenDataset (train): 520,865 sequences × 2048 tokens = 1.07B tokens total [2026-10-21 06:49:30][INFO][__main__] Loaded 'redpajama_cc': 520,865 sequences [2026-10-21 06:49:30][INFO][data.dataset] PackedTokenDataset (train): 973,844 sequences × 2048 tokens = 1.99B tokens total [2026-10-21 06:49:30][INFO][__main__] Loaded 'multilingual': 973,844 sequences [2026-10-21 06:49:30][INFO][data.dataset] PackedTokenDataset (train): 973,844 sequences × 2048 tokens = 1.99B tokens total [2026-10-21 06:49:30][INFO][__main__] Loaded 'multilingual': 973,844 sequences [2026-10-21 06:49:30][INFO][data.dataset] PackedTokenDataset (train): 3,202,429 sequences × 2048 tokens = 6.56B tokens total [2026-10-21 06:49:30][INFO][__main__] Loaded 'arxiv': 3,202,429 sequences [2026-10-21 06:49:30][INFO][data.dataset] PackedTokenDataset (train): 3,202,429 sequences × 2048 tokens = 6.56B tokens total [2026-10-21 06:49:30][INFO][__main__] Loaded 'arxiv': 3,202,429 sequences [2026-10-21 06:49:30][INFO][data.dataset] PackedTokenDataset (train): 2,418,666 sequences × 2048 tokens = 4.95B tokens total [2026-10-21 06:49:30][INFO][__main__] Loaded 'dolma': 2,418,666 sequences [2026-10-21 06:49:30][INFO][data.dataset] PackedTokenDataset (train): 2,418,666 sequences × 2048 tokens = 4.95B tokens total [2026-10-21 06:49:30][INFO][__main__] Loaded 'dolma': 2,418,666 sequences [2026-10-21 06:49:30][INFO][data.dataset] PackedTokenDataset (train): 1,178,010 sequences × 2048 tokens = 2.41B tokens total [2026-10-21 06:49:30][INFO][__main__] Loaded 'stackexchange': 1,178,010 sequences [2026-10-21 06:49:30][INFO][data.dataset] PackedTokenDataset (train): 1,178,010 sequences × 2048 tokens = 2.41B tokens total [2026-10-21 06:49:30][INFO][__main__] Loaded 'stackexchange': 1,178,010 sequences [2026-10-21 06:49:30][INFO][data.dataset] PackedTokenDataset (train): 520,865 sequences × 2048 tokens = 1.07B tokens total [2026-10-21 06:49:30][INFO][__main__] Loaded 'redpajama_cc': 520,865 sequences [2026-10-21 06:49:30][INFO][data.dataset] PackedTokenDataset (train): 520,865 sequences × 2048 tokens = 1.07B tokens total [2026-10-21 06:49:30][INFO][__main__] Loaded 'redpajama_cc': 520,865 sequences [2026-10-21 06:49:30][INFO][data.dataset] PackedTokenDataset (train): 973,844 sequences × 2048 tokens = 1.99B tokens total [2026-10-21 06:49:30][INFO][__main__] Loaded 'multilingual': 973,844 sequences [2026-10-21 06:49:30][INFO][data.dataset] PackedTokenDataset (train): 973,844 sequences × 2048 tokens = 1.99B tokens total [2026-10-21 06:49:30][INFO][__main__] Loaded 'multilingual': 973,844 sequences [2026-10-21 06:49:30][INFO][data.dataset] WeightedDataMixer: 7,324,218 total samples from 10 sources with weights {'fineweb_edu': 0.3, 'thestack': 0.16, 'wikipedia': 0.08, 'openwebmath': 0.08, 'books': 0.07, 'arxiv': 0.06, 'dolma': 0.1, 'stackexchange': 0.05, 'redpajama_cc': 0.06, 'multilingual': 0.04} [2026-10-21 06:49:30][INFO][data.dataset] WeightedDataMixer: 7,324,218 total samples from 10 sources with weights {'fineweb_edu': 0.3, 'thestack': 0.16, 'wikipedia': 0.08, 'openwebmath': 0.08, 'books': 0.07, 'arxiv': 0.06, 'dolma': 0.1, 'stackexchange': 0.05, 'redpajama_cc': 0.06, 'multilingual': 0.04} [2026-10-21 06:49:30][INFO][__main__] Phase-2 loader ready: 7,324,218 sequences (batch_size=4), grad_accum=32 [2026-10-21 06:49:30][INFO][__main__] Phase 3 disabled (phase3_tokens=0) — pure seq=2048 pretraining. [2026-10-21 06:49:30][INFO][__main__] Building AdamW optimizer with parameter-group differentiation... [2026-10-21 06:49:30][INFO][__main__] Phase-2 loader ready: 7,324,218 sequences (batch_size=4), grad_accum=32 [2026-10-21 06:49:30][INFO][__main__] Phase 3 disabled (phase3_tokens=0) — pure seq=2048 pretraining. [2026-10-21 06:49:30][INFO][__main__] Building AdamW optimizer with parameter-group differentiation... [2026-10-21 06:49:30][INFO][training.optimizer] Optimizer group 'main': 287.9M params, lr=3.00e-04, wd=0.1 [2026-10-21 06:49:30][INFO][training.optimizer] Optimizer group 'no_decay': 0.1M params, lr=3.00e-04, wd=0.0 [2026-10-21 06:49:30][INFO][training.optimizer] Optimizer group 'embedding': 211.4M params, lr=1.50e-04, wd=0.0 [2026-10-21 06:49:30][INFO][training.optimizer] Optimizer group 'ssm_state': 0.0M params, lr=9.00e-05, wd=0.0 [2026-10-21 06:49:30][INFO][training.optimizer] Optimizer group 'moe_router': 0.3M params, lr=1.50e-04, wd=0.0 [2026-10-21 06:49:30][INFO][training.optimizer] Optimizer group 'main': 287.9M params, lr=3.00e-04, wd=0.1 [2026-10-21 06:49:30][INFO][training.optimizer] Optimizer group 'no_decay': 0.1M params, lr=3.00e-04, wd=0.0 [2026-10-21 06:49:30][INFO][training.optimizer] Optimizer group 'embedding': 211.4M params, lr=1.50e-04, wd=0.0 [2026-10-21 06:49:30][INFO][training.optimizer] Optimizer group 'ssm_state': 0.0M params, lr=9.00e-05, wd=0.0 [2026-10-21 06:49:30][INFO][training.optimizer] Optimizer group 'moe_router': 0.3M params, lr=1.50e-04, wd=0.0 [2026-10-21 06:49:30][INFO][training.optimizer] Using fused AdamW (CUDA native) [2026-10-21 06:49:30][INFO][training.optimizer] Using fused AdamW (CUDA native) [2026-10-21 06:49:30][INFO][data.dataset] WeightedDataMixer: 7,324,218 total samples from 10 sources with weights {'fineweb_edu': 0.3, 'thestack': 0.16, 'wikipedia': 0.08, 'openwebmath': 0.08, 'books': 0.07, 'arxiv': 0.06, 'dolma': 0.1, 'stackexchange': 0.05, 'redpajama_cc': 0.06, 'multilingual': 0.04} [2026-10-21 06:49:30][INFO][__main__] Phase-2 loader ready: 7,324,218 sequences (batch_size=4), grad_accum=32 [2026-10-21 06:49:30][INFO][__main__] Phase 3 disabled (phase3_tokens=0) — pure seq=2048 pretraining. [2026-10-21 06:49:30][INFO][__main__] Building AdamW optimizer with parameter-group differentiation... [2026-10-21 06:49:30][INFO][training.optimizer] Optimizer group 'main': 287.9M params, lr=3.00e-04, wd=0.1 [2026-10-21 06:49:30][INFO][training.optimizer] Optimizer group 'no_decay': 0.1M params, lr=3.00e-04, wd=0.0 [2026-10-21 06:49:30][INFO][training.optimizer] Optimizer group 'embedding': 211.4M params, lr=1.50e-04, wd=0.0 [2026-10-21 06:49:30][INFO][training.optimizer] Optimizer group 'ssm_state': 0.0M params, lr=9.00e-05, wd=0.0 [2026-10-21 06:49:30][INFO][training.optimizer] Optimizer group 'moe_router': 0.3M params, lr=1.50e-04, wd=0.0 [2026-10-21 06:49:30][INFO][training.optimizer] Using fused AdamW (CUDA native) [2026-10-21 06:49:30][INFO][data.dataset] WeightedDataMixer: 7,324,218 total samples from 10 sources with weights {'fineweb_edu': 0.3, 'thestack': 0.16, 'wikipedia': 0.08, 'openwebmath': 0.08, 'books': 0.07, 'arxiv': 0.06, 'dolma': 0.1, 'stackexchange': 0.05, 'redpajama_cc': 0.06, 'multilingual': 0.04} [2026-10-21 06:49:30][INFO][__main__] Phase-2 loader ready: 7,324,218 sequences (batch_size=4), grad_accum=32 [2026-10-21 06:49:30][INFO][__main__] Phase 3 disabled (phase3_tokens=0) — pure seq=2048 pretraining. [2026-10-21 06:49:30][INFO][__main__] Building AdamW optimizer with parameter-group differentiation... [2026-10-21 06:49:30][INFO][training.optimizer] Optimizer group 'main': 287.9M params, lr=3.00e-04, wd=0.1 [2026-10-21 06:49:30][INFO][training.optimizer] Optimizer group 'no_decay': 0.1M params, lr=3.00e-04, wd=0.0 [2026-10-21 06:49:30][INFO][training.optimizer] Optimizer group 'embedding': 211.4M params, lr=1.50e-04, wd=0.0 [2026-10-21 06:49:30][INFO][training.optimizer] Optimizer group 'ssm_state': 0.0M params, lr=9.00e-05, wd=0.0 [2026-10-21 06:49:30][INFO][training.optimizer] Optimizer group 'moe_router': 0.3M params, lr=1.50e-04, wd=0.0 [2026-10-21 06:49:30][INFO][training.optimizer] Using fused AdamW (CUDA native) [2026-10-21 06:49:31][INFO][__main__] ✅ Optimizer state restored exactly from checkpoint. [2026-10-21 06:49:31][INFO][__main__] ✅ Scheduler state restored exactly from checkpoint. [2026-10-21 06:49:31][INFO][__main__] ✅ Optimizer state restored exactly from checkpoint. [2026-10-21 06:49:31][INFO][__main__] ✅ Scheduler state restored exactly from checkpoint. [2026-10-21 06:49:31][INFO][__main__] ✅ Optimizer state restored exactly from checkpoint. [2026-10-21 06:49:31][INFO][__main__] ✅ Scheduler state restored exactly from checkpoint. [2026-10-21 06:49:31][INFO][__main__] ✅ Optimizer state restored exactly from checkpoint. [2026-10-21 06:49:31][INFO][__main__] ✅ Scheduler state restored exactly from checkpoint. [2026-10-21 06:49:31][INFO][training.gpu_optimizer] GPU Memory[pre-training] step=0: 2.44GB alloc / 2.46GB reserved / 30.29GB free / 32.75GB total [2026-10-21 06:49:31][INFO][training.gpu_optimizer] GPU Memory[pre-training] step=0: 2.44GB alloc / 2.46GB reserved / 30.29GB free / 32.75GB total [2026-10-21 06:49:31][INFO][training.gpu_optimizer] GPU Memory[pre-training] step=0: 2.44GB alloc / 2.46GB reserved / 30.29GB free / 32.75GB total [2026-10-21 06:49:31][INFO][training.gpu_optimizer] GPU Memory[pre-training] step=0: 2.44GB alloc / 2.46GB reserved / 30.29GB free / 32.75GB total [2026-10-21 06:49:31][INFO][__main__] Initializing PretrainEngine... [2026-10-21 06:49:31][INFO][__main__] Initializing PretrainEngine... [2026-10-21 06:49:31][INFO][__main__] Initializing PretrainEngine... [2026-10-21 06:49:31][INFO][__main__] Initializing PretrainEngine... [2026-10-21 06:49:31][INFO][training.gpu_optimizer] GPU detected: NVIDIA B300 SXM6 AC MIG 1g.34gb (MIG 18/144 SMs) — 312.5 TFLOPS BF16 (MIG slice peak) [2026-10-21 06:49:31][INFO][training.gpu_optimizer] GPU detected: NVIDIA B300 SXM6 AC MIG 1g.34gb (MIG 18/144 SMs) — 312.5 TFLOPS BF16 (MIG slice peak) [2026-10-21 06:49:31][INFO][__main__] Engine token counter initialised to 0.80B [2026-10-21 06:49:31][INFO][__main__] Engine token counter initialised to 0.80B [2026-10-21 06:49:31][INFO][training.gpu_optimizer] Warming up GPU cuda:1... [2026-10-21 06:49:31][INFO][__main__] ====================================================================== [2026-10-21 06:49:31][INFO][__main__] FORGE-3B PRETRAINING — FULL THREE-PHASE RUN [2026-10-21 06:49:31][INFO][__main__] Phase 1 : 0B tokens | seq=512 [2026-10-21 06:49:31][INFO][__main__] Phase 2 : 15B tokens | seq=2048 [2026-10-21 06:49:31][INFO][__main__] Phase 3 : 0B tokens | seq=2048 [2026-10-21 06:49:31][INFO][__main__] Total : 15B tokens [2026-10-21 06:49:31][INFO][__main__] Budget : $450.00 | Rate: $63.17/hr (16× H100 SXM) [2026-10-21 06:49:31][INFO][__main__] ====================================================================== [2026-10-21 06:49:31][INFO][training.gpu_optimizer] Warming up GPU cuda:0... [2026-10-21 06:49:31][INFO][training.gpu_optimizer] GPU detected: NVIDIA B300 SXM6 AC MIG 1g.34gb (MIG 18/144 SMs) — 312.5 TFLOPS BF16 (MIG slice peak) [2026-10-21 06:49:31][INFO][__main__] Engine token counter initialised to 0.80B [2026-10-21 06:49:31][INFO][training.gpu_optimizer] Warming up GPU cuda:2... [2026-10-21 06:49:31][INFO][training.gpu_optimizer] GPU detected: NVIDIA B300 SXM6 AC MIG 1g.34gb (MIG 18/144 SMs) — 312.5 TFLOPS BF16 (MIG slice peak) [2026-10-21 06:49:31][INFO][__main__] Engine token counter initialised to 0.80B [2026-10-21 06:49:31][INFO][training.gpu_optimizer] Warming up GPU cuda:3... [2026-10-21 06:49:41][INFO][training.gpu_optimizer] GPU cuda:1 warmed up. Memory: 2.47GB allocated, 2.50GB reserved /workspace/forge_3b/training/pretrain_engine.py:106: FutureWarning: `broadcast_buffers` is deprecated. Use `forward_sync_buffers` instead. IMPORTANT: unlike `broadcast_buffers=False`, `forward_sync_buffers=False` still syncs buffers at init. If you rely on buffers NOT being synced at init (e.g. rank-local buffers), keep using `broadcast_buffers=False` until `init_sync_buffers` is available. self.model = DDP( [2026-10-21 06:49:41][INFO][training.gpu_optimizer] GPU cuda:0 warmed up. Memory: 2.47GB allocated, 2.50GB reserved /workspace/forge_3b/training/pretrain_engine.py:106: FutureWarning: `broadcast_buffers` is deprecated. Use `forward_sync_buffers` instead. IMPORTANT: unlike `broadcast_buffers=False`, `forward_sync_buffers=False` still syncs buffers at init. If you rely on buffers NOT being synced at init (e.g. rank-local buffers), keep using `broadcast_buffers=False` until `init_sync_buffers` is available. self.model = DDP( [2026-10-21 06:49:41][INFO][training.gpu_optimizer] GPU cuda:2 warmed up. Memory: 2.47GB allocated, 2.50GB reserved /workspace/forge_3b/training/pretrain_engine.py:106: FutureWarning: `broadcast_buffers` is deprecated. Use `forward_sync_buffers` instead. IMPORTANT: unlike `broadcast_buffers=False`, `forward_sync_buffers=False` still syncs buffers at init. If you rely on buffers NOT being synced at init (e.g. rank-local buffers), keep using `broadcast_buffers=False` until `init_sync_buffers` is available. self.model = DDP( [2026-10-21 06:49:41][INFO][training.gpu_optimizer] GPU cuda:3 warmed up. Memory: 2.47GB allocated, 2.50GB reserved /workspace/forge_3b/training/pretrain_engine.py:106: FutureWarning: `broadcast_buffers` is deprecated. Use `forward_sync_buffers` instead. IMPORTANT: unlike `broadcast_buffers=False`, `forward_sync_buffers=False` still syncs buffers at init. If you rely on buffers NOT being synced at init (e.g. rank-local buffers), keep using `broadcast_buffers=False` until `init_sync_buffers` is available. self.model = DDP( [2026-10-21 06:49:42][INFO][training.pretrain_engine] 🚀 Wrapped model in DDP (world_size=4, static_graph=False, find_unused_parameters=True) [2026-10-21 06:49:42][INFO][training.pretrain_engine] ====================================================================== [2026-10-21 06:49:42][INFO][training.pretrain_engine] FORGE-500M PRETRAINING STARTED (plain PyTorch DDP, no DeepSpeed) [2026-10-21 06:49:42][INFO][training.pretrain_engine] Target: 15B tokens (Chinchilla-optimal for 500M) [2026-10-21 06:49:42][INFO][training.pretrain_engine] Budget: $1000 | Cost rate: $2.06/hr [2026-10-21 06:49:42][INFO][training.pretrain_engine] Max training hours: 485.4h [2026-10-21 06:49:42][INFO][training.pretrain_engine] World size: 4 [2026-10-21 06:49:42][INFO][training.pretrain_engine] ====================================================================== [2026-10-21 06:49:42][INFO][training.pretrain_engine] ================================================== [2026-10-21 06:49:42][INFO][training.pretrain_engine] PHASE 1: SKIPPED (Starting directly on Phase 2: Core Pretraining @ seq=2048) [2026-10-21 06:49:42][INFO][training.pretrain_engine] ARG local window updated to 64 [2026-10-21 06:49:42][INFO][training.pretrain_engine] ================================================== [2026-10-21 06:49:42][INFO][training.pretrain_engine] PHASE 2: Core Pre-Training (15.0B tokens, seq=2048) [2026-10-21 06:49:42][INFO][training.pretrain_engine] 🚀 Wrapped model in DDP (world_size=4, static_graph=False, find_unused_parameters=True) [2026-10-21 06:49:42][INFO][training.pretrain_engine] ====================================================================== [2026-10-21 06:49:42][INFO][training.pretrain_engine] FORGE-500M PRETRAINING STARTED (plain PyTorch DDP, no DeepSpeed) [2026-10-21 06:49:42][INFO][training.pretrain_engine] Target: 15B tokens (Chinchilla-optimal for 500M) [2026-10-21 06:49:42][INFO][training.pretrain_engine] Budget: $1000 | Cost rate: $2.06/hr [2026-10-21 06:49:42][INFO][training.pretrain_engine] Max training hours: 485.4h [2026-10-21 06:49:42][INFO][training.pretrain_engine] World size: 4 [2026-10-21 06:49:42][INFO][training.pretrain_engine] ====================================================================== [2026-10-21 06:49:42][INFO][training.pretrain_engine] ================================================== [2026-10-21 06:49:42][INFO][training.pretrain_engine] PHASE 1: SKIPPED (Starting directly on Phase 2: Core Pretraining @ seq=2048) [2026-10-21 06:49:42][INFO][training.pretrain_engine] ARG local window updated to 64 [2026-10-21 06:49:42][INFO][training.pretrain_engine] ================================================== [2026-10-21 06:49:42][INFO][training.pretrain_engine] PHASE 2: Core Pre-Training (15.0B tokens, seq=2048) [2026-10-21 06:49:42][INFO][training.pretrain_engine] 🚀 Wrapped model in DDP (world_size=4, static_graph=False, find_unused_parameters=True) [2026-10-21 06:49:42][INFO][training.pretrain_engine] ====================================================================== [2026-10-21 06:49:42][INFO][training.pretrain_engine] FORGE-500M PRETRAINING STARTED (plain PyTorch DDP, no DeepSpeed) [2026-10-21 06:49:42][INFO][training.pretrain_engine] Target: 15B tokens (Chinchilla-optimal for 500M) [2026-10-21 06:49:42][INFO][training.pretrain_engine] Budget: $1000 | Cost rate: $2.06/hr [2026-10-21 06:49:42][INFO][training.pretrain_engine] Max training hours: 485.4h [2026-10-21 06:49:42][INFO][training.pretrain_engine] World size: 4 [2026-10-21 06:49:42][INFO][training.pretrain_engine] ====================================================================== [2026-10-21 06:49:42][INFO][training.pretrain_engine] ================================================== [2026-10-21 06:49:42][INFO][training.pretrain_engine] PHASE 1: SKIPPED (Starting directly on Phase 2: Core Pretraining @ seq=2048) [2026-10-21 06:49:42][INFO][training.pretrain_engine] ARG local window updated to 64 [2026-10-21 06:49:42][INFO][training.pretrain_engine] 🚀 Wrapped model in DDP (world_size=4, static_graph=False, find_unused_parameters=True) [2026-10-21 06:49:42][INFO][training.pretrain_engine] ================================================== [2026-10-21 06:49:42][INFO][training.pretrain_engine] PHASE 2: Core Pre-Training (15.0B tokens, seq=2048) [2026-10-21 06:49:42][INFO][training.pretrain_engine] ====================================================================== [2026-10-21 06:49:42][INFO][training.pretrain_engine] FORGE-500M PRETRAINING STARTED (plain PyTorch DDP, no DeepSpeed) [2026-10-21 06:49:42][INFO][training.pretrain_engine] Target: 15B tokens (Chinchilla-optimal for 500M) [2026-10-21 06:49:42][INFO][training.pretrain_engine] Budget: $1000 | Cost rate: $2.06/hr [2026-10-21 06:49:42][INFO][training.pretrain_engine] Max training hours: 485.4h [2026-10-21 06:49:42][INFO][training.pretrain_engine] World size: 4 [2026-10-21 06:49:42][INFO][training.pretrain_engine] ====================================================================== [2026-10-21 06:49:42][INFO][training.pretrain_engine] ================================================== [2026-10-21 06:49:42][INFO][training.pretrain_engine] PHASE 1: SKIPPED (Starting directly on Phase 2: Core Pretraining @ seq=2048) [2026-10-21 06:49:42][INFO][training.pretrain_engine] ARG local window updated to 64 [2026-10-21 06:49:42][INFO][training.pretrain_engine] ================================================== [2026-10-21 06:49:42][INFO][training.pretrain_engine] PHASE 2: Core Pre-Training (15.0B tokens, seq=2048) [2026-10-21 06:49:42][INFO][training.pretrain_engine] Starting global step 1 (accumulating 32 micro-batches)... [2026-10-21 06:49:42][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 06:50:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4779 [2026-10-21 06:50:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0492 [2026-10-21 06:50:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5094 [2026-10-21 06:50:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0862 [2026-10-21 06:50:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2214 [2026-10-21 06:50:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3844 [2026-10-21 06:50:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6923 [2026-10-21 06:50:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3696 [2026-10-21 06:51:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2188 [2026-10-21 06:51:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1974 [2026-10-21 06:51:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6174 [2026-10-21 06:51:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7439 [2026-10-21 06:51:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4414 [2026-10-21 06:51:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4159 [2026-10-21 06:51:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7839 [2026-10-21 06:51:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5158 [2026-10-21 06:51:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9728 [2026-10-21 06:51:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9716 [2026-10-21 06:51:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1954 [2026-10-21 06:51:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4892 [2026-10-21 06:51:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0702 [2026-10-21 06:51:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3762 [2026-10-21 06:51:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5094 [2026-10-21 06:51:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5287 [2026-10-21 06:52:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4472 [2026-10-21 06:52:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6903 [2026-10-21 06:52:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2815 [2026-10-21 06:52:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0405 [2026-10-21 06:52:28][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 6,328 | MFU: 1.5% [2026-10-21 06:52:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0748 [2026-10-21 06:52:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5171 [2026-10-21 06:52:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1733 [2026-10-21 06:52:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7607 [2026-10-21 06:52:40][INFO][training.pretrain_engine] [Ph2|Step 1] loss=5.3544 | ppl=211.55 | aux=0.02508 | lr=3.00e-04 | gnorm=0.586 | tok/s=23,556 | MFU=5.7% | step=178057ms | tokens=0.803B | mem=4.2/32.7GB | cost=$0.11/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 28 06:17 (167.4h) | Phase3: SKIPPED | SFT: Oct 28 12:11 (173.3h) | DPO (Final): Oct 28 12:53 (174.0h) [2026-10-21 06:52:40][INFO][training.pretrain_engine] Starting global step 2 (accumulating 32 micro-batches)... [2026-10-21 06:52:40][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 06:52:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4010 [2026-10-21 06:52:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8351 [2026-10-21 06:52:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6885 [2026-10-21 06:52:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5088 [2026-10-21 06:53:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4672 [2026-10-21 06:53:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5353 [2026-10-21 06:53:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5344 [2026-10-21 06:53:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6381 [2026-10-21 06:53:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3386 [2026-10-21 06:53:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5811 [2026-10-21 06:53:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7897 [2026-10-21 06:53:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8850 [2026-10-21 06:53:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4931 [2026-10-21 06:53:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3927 [2026-10-21 06:53:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6952 [2026-10-21 06:53:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8664 [2026-10-21 06:54:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4441 [2026-10-21 06:54:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8803 [2026-10-21 06:54:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5126 [2026-10-21 06:54:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5414 [2026-10-21 06:54:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5381 [2026-10-21 06:54:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5788 [2026-10-21 06:54:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2116 [2026-10-21 06:54:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3143 [2026-10-21 06:54:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9845 [2026-10-21 06:54:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5862 [2026-10-21 06:54:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6347 [2026-10-21 06:54:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8876 [2026-10-21 06:54:58][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,610 | MFU: 1.8% [2026-10-21 06:55:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9199 [2026-10-21 06:55:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3404 [2026-10-21 06:55:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5401 [2026-10-21 06:55:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.4490 [2026-10-21 06:55:02][INFO][training.pretrain_engine] [Ph2|Step 2] loss=5.4178 | ppl=225.39 | aux=0.02507 | lr=3.00e-04 | gnorm=0.559 | tok/s=26,183 | MFU=6.3% | step=160192ms | tokens=0.804B | mem=4.2/32.7GB | cost=$0.19/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 13:31 (150.6h) | Phase3: SKIPPED | SFT: Oct 27 18:49 (155.9h) | DPO (Final): Oct 27 19:27 (156.5h) [2026-10-21 06:55:02][INFO][training.pretrain_engine] Starting global step 3 (accumulating 32 micro-batches)... [2026-10-21 06:55:02][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 06:55:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7996 [2026-10-21 06:55:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9833 [2026-10-21 06:55:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2548 [2026-10-21 06:55:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2931 [2026-10-21 06:55:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8706 [2026-10-21 06:55:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7760 [2026-10-21 06:55:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7706 [2026-10-21 06:55:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3475 [2026-10-21 06:55:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1754 [2026-10-21 06:55:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7100 [2026-10-21 06:55:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8727 [2026-10-21 06:55:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4329 [2026-10-21 06:56:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3276 [2026-10-21 06:56:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1994 [2026-10-21 06:56:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3924 [2026-10-21 06:56:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6101 [2026-10-21 06:56:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9910 [2026-10-21 06:56:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3850 [2026-10-21 06:56:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5409 [2026-10-21 06:56:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4336 [2026-10-21 06:56:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3837 [2026-10-21 06:56:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2173 [2026-10-21 06:56:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5583 [2026-10-21 06:56:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3642 [2026-10-21 06:57:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4655 [2026-10-21 06:57:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1242 [2026-10-21 06:57:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4763 [2026-10-21 06:57:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8427 [2026-10-21 06:57:20][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,630 | MFU: 1.8% [2026-10-21 06:57:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3034 [2026-10-21 06:57:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6719 [2026-10-21 06:57:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2249 [2026-10-21 06:57:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5721 [2026-10-21 06:57:25][INFO][training.pretrain_engine] [Ph2|Step 3] loss=5.4773 | ppl=239.20 | aux=0.02508 | lr=3.00e-04 | gnorm=0.613 | tok/s=27,189 | MFU=6.5% | step=154264ms | tokens=0.805B | mem=4.2/32.7GB | cost=$0.27/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 07:58 (145.0h) | Phase3: SKIPPED | SFT: Oct 27 13:05 (150.1h) | DPO (Final): Oct 27 13:41 (150.7h) [2026-10-21 06:57:25][INFO][training.pretrain_engine] Starting global step 4 (accumulating 32 micro-batches)... [2026-10-21 06:57:25][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 06:57:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3079 [2026-10-21 06:57:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3728 [2026-10-21 06:57:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9324 [2026-10-21 06:57:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2599 [2026-10-21 06:58:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9589 [2026-10-21 06:58:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2089 [2026-10-21 06:58:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2996 [2026-10-21 06:58:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3695 [2026-10-21 06:58:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1765 [2026-10-21 06:58:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4086 [2026-10-21 06:58:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6512 [2026-10-21 06:58:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5531 [2026-10-21 06:58:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4004 [2026-10-21 06:58:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2284 [2026-10-21 06:58:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4525 [2026-10-21 06:58:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4289 [2026-10-21 06:58:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8996 [2026-10-21 06:58:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7191 [2026-10-21 06:58:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3834 [2026-10-21 06:58:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7321 [2026-10-21 06:59:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7072 [2026-10-21 06:59:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6627 [2026-10-21 06:59:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7911 [2026-10-21 06:59:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6176 [2026-10-21 06:59:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5473 [2026-10-21 06:59:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5092 [2026-10-21 06:59:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2308 [2026-10-21 06:59:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1258 [2026-10-21 06:59:42][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,626 | MFU: 1.8% [2026-10-21 06:59:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2834 [2026-10-21 06:59:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4464 [2026-10-21 06:59:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5044 [2026-10-21 06:59:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5071 [2026-10-21 06:59:47][INFO][training.pretrain_engine] [Ph2|Step 4] loss=5.4103 | ppl=223.69 | aux=0.02508 | lr=3.00e-04 | gnorm=0.605 | tok/s=27,717 | MFU=6.7% | step=151324ms | tokens=0.806B | mem=4.2/32.7GB | cost=$0.35/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 05:14 (142.2h) | Phase3: SKIPPED | SFT: Oct 27 10:15 (147.3h) | DPO (Final): Oct 27 10:51 (147.9h) [2026-10-21 06:59:47][INFO][training.pretrain_engine] Starting global step 5 (accumulating 32 micro-batches)... [2026-10-21 06:59:47][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 07:00:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0958 [2026-10-21 07:00:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1923 [2026-10-21 07:00:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3293 [2026-10-21 07:00:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6665 [2026-10-21 07:00:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1183 [2026-10-21 07:00:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1223 [2026-10-21 07:00:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9325 [2026-10-21 07:00:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0737 [2026-10-21 07:00:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4020 [2026-10-21 07:00:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7990 [2026-10-21 07:00:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7888 [2026-10-21 07:00:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3173 [2026-10-21 07:00:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2221 [2026-10-21 07:00:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0141 [2026-10-21 07:00:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5867 [2026-10-21 07:00:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7450 [2026-10-21 07:01:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4395 [2026-10-21 07:01:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4246 [2026-10-21 07:01:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2662 [2026-10-21 07:01:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1381 [2026-10-21 07:01:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8904 [2026-10-21 07:01:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5867 [2026-10-21 07:01:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4915 [2026-10-21 07:01:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2665 [2026-10-21 07:01:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5168 [2026-10-21 07:01:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=3.2892 [2026-10-21 07:01:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1610 [2026-10-21 07:01:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9622 [2026-10-21 07:02:05][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,621 | MFU: 1.8% [2026-10-21 07:02:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3332 [2026-10-21 07:02:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0894 [2026-10-21 07:02:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0384 [2026-10-21 07:02:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5671 [2026-10-21 07:02:10][INFO][training.pretrain_engine] [Ph2|Step 5] loss=5.4254 | ppl=227.11 | aux=0.02509 | lr=3.00e-04 | gnorm=0.543 | tok/s=28,044 | MFU=6.7% | step=149563ms | tokens=0.807B | mem=4.2/32.7GB | cost=$0.43/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 03:36 (140.6h) | Phase3: SKIPPED | SFT: Oct 27 08:34 (145.5h) | DPO (Final): Oct 27 09:09 (146.1h) [2026-10-21 07:02:10][INFO][training.pretrain_engine] Starting global step 6 (accumulating 32 micro-batches)... [2026-10-21 07:02:10][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 07:02:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2212 [2026-10-21 07:02:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6284 [2026-10-21 07:02:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7513 [2026-10-21 07:02:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9615 [2026-10-21 07:02:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4281 [2026-10-21 07:02:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6773 [2026-10-21 07:02:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9831 [2026-10-21 07:02:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5908 [2026-10-21 07:03:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4017 [2026-10-21 07:03:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6982 [2026-10-21 07:03:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8450 [2026-10-21 07:03:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3078 [2026-10-21 07:03:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1747 [2026-10-21 07:03:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7639 [2026-10-21 07:03:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1960 [2026-10-21 07:03:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6946 [2026-10-21 07:03:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2220 [2026-10-21 07:03:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2388 [2026-10-21 07:03:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1273 [2026-10-21 07:03:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9095 [2026-10-21 07:03:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6909 [2026-10-21 07:03:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3555 [2026-10-21 07:03:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6512 [2026-10-21 07:03:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9912 [2026-10-21 07:04:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0792 [2026-10-21 07:04:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1754 [2026-10-21 07:04:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5447 [2026-10-21 07:04:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1261 [2026-10-21 07:04:27][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,627 | MFU: 1.8% [2026-10-21 07:04:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2408 [2026-10-21 07:04:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2731 [2026-10-21 07:04:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0617 [2026-10-21 07:04:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4278 [2026-10-21 07:04:32][INFO][training.pretrain_engine] [Ph2|Step 6] loss=5.4625 | ppl=235.68 | aux=0.02509 | lr=3.00e-04 | gnorm=0.703 | tok/s=28,274 | MFU=6.8% | step=148345ms | tokens=0.808B | mem=4.2/32.7GB | cost=$0.52/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 02:30 (139.4h) | Phase3: SKIPPED | SFT: Oct 27 07:24 (144.3h) | DPO (Final): Oct 27 08:00 (144.9h) [2026-10-21 07:04:32][INFO][training.pretrain_engine] Starting global step 7 (accumulating 32 micro-batches)... [2026-10-21 07:04:32][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 07:04:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4935 [2026-10-21 07:04:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1259 [2026-10-21 07:04:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7239 [2026-10-21 07:04:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9686 [2026-10-21 07:05:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7696 [2026-10-21 07:05:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5192 [2026-10-21 07:05:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5166 [2026-10-21 07:05:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1909 [2026-10-21 07:05:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9282 [2026-10-21 07:05:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8258 [2026-10-21 07:05:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4600 [2026-10-21 07:05:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4389 [2026-10-21 07:05:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3137 [2026-10-21 07:05:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2414 [2026-10-21 07:05:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1777 [2026-10-21 07:05:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0622 [2026-10-21 07:06:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0825 [2026-10-21 07:06:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8880 [2026-10-21 07:06:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7888 [2026-10-21 07:06:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4357 [2026-10-21 07:06:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4567 [2026-10-21 07:06:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3817 [2026-10-21 07:06:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4374 [2026-10-21 07:06:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2616 [2026-10-21 07:06:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4379 [2026-10-21 07:06:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7203 [2026-10-21 07:06:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7289 [2026-10-21 07:06:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4573 [2026-10-21 07:06:50][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,632 | MFU: 1.8% [2026-10-21 07:06:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5354 [2026-10-21 07:06:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4709 [2026-10-21 07:06:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6659 [2026-10-21 07:06:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5174 [2026-10-21 07:06:55][INFO][training.pretrain_engine] [Ph2|Step 7] loss=5.4270 | ppl=227.47 | aux=0.02508 | lr=3.00e-04 | gnorm=0.770 | tok/s=28,430 | MFU=6.8% | step=147532ms | tokens=0.810B | mem=4.2/32.7GB | cost=$0.60/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 01:45 (138.7h) | Phase3: SKIPPED | SFT: Oct 27 06:39 (143.5h) | DPO (Final): Oct 27 07:14 (144.1h) [2026-10-21 07:06:55][INFO][training.pretrain_engine] Starting global step 8 (accumulating 32 micro-batches)... [2026-10-21 07:06:55][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 07:07:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4711 [2026-10-21 07:07:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9998 [2026-10-21 07:07:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8192 [2026-10-21 07:07:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1874 [2026-10-21 07:07:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5656 [2026-10-21 07:07:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3363 [2026-10-21 07:07:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5093 [2026-10-21 07:07:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7929 [2026-10-21 07:07:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3870 [2026-10-21 07:07:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4349 [2026-10-21 07:07:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3512 [2026-10-21 07:07:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5121 [2026-10-21 07:08:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9014 [2026-10-21 07:08:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2870 [2026-10-21 07:08:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2052 [2026-10-21 07:08:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6289 [2026-10-21 07:08:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4165 [2026-10-21 07:08:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2934 [2026-10-21 07:08:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7898 [2026-10-21 07:08:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8565 [2026-10-21 07:08:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8584 [2026-10-21 07:08:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7964 [2026-10-21 07:08:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9466 [2026-10-21 07:08:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7755 [2026-10-21 07:08:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6341 [2026-10-21 07:08:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5716 [2026-10-21 07:08:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6538 [2026-10-21 07:09:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7867 [2026-10-21 07:09:13][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,608 | MFU: 1.8% [2026-10-21 07:09:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5734 [2026-10-21 07:09:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2873 [2026-10-21 07:09:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6247 [2026-10-21 07:09:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4144 [2026-10-21 07:09:18][INFO][training.pretrain_engine] [Ph2|Step 8] loss=5.5303 | ppl=252.21 | aux=0.02507 | lr=3.00e-04 | gnorm=0.980 | tok/s=28,544 | MFU=6.9% | step=146942ms | tokens=0.811B | mem=4.2/32.7GB | cost=$0.68/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 01:14 (138.1h) | Phase3: SKIPPED | SFT: Oct 27 06:06 (143.0h) | DPO (Final): Oct 27 06:41 (143.5h) [2026-10-21 07:09:18][INFO][training.pretrain_engine] Starting global step 9 (accumulating 32 micro-batches)... [2026-10-21 07:09:18][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 07:09:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4653 [2026-10-21 07:09:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1531 [2026-10-21 07:09:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0997 [2026-10-21 07:09:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0056 [2026-10-21 07:09:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4256 [2026-10-21 07:09:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2673 [2026-10-21 07:09:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1007 [2026-10-21 07:09:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3987 [2026-10-21 07:10:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6890 [2026-10-21 07:10:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9995 [2026-10-21 07:10:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2089 [2026-10-21 07:10:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8323 [2026-10-21 07:10:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3991 [2026-10-21 07:10:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3621 [2026-10-21 07:10:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5306 [2026-10-21 07:10:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2675 [2026-10-21 07:10:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7564 [2026-10-21 07:10:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8084 [2026-10-21 07:10:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6800 [2026-10-21 07:10:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3866 [2026-10-21 07:11:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8044 [2026-10-21 07:11:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0975 [2026-10-21 07:11:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7803 [2026-10-21 07:11:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4271 [2026-10-21 07:11:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7874 [2026-10-21 07:11:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3802 [2026-10-21 07:11:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1446 [2026-10-21 07:11:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6102 [2026-10-21 07:11:35][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,612 | MFU: 1.8% [2026-10-21 07:11:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4589 [2026-10-21 07:11:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4710 [2026-10-21 07:11:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3750 [2026-10-21 07:11:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4588 [2026-10-21 07:11:40][INFO][training.pretrain_engine] [Ph2|Step 9] loss=5.4015 | ppl=221.73 | aux=0.02509 | lr=3.00e-04 | gnorm=1.320 | tok/s=28,646 | MFU=6.9% | step=146418ms | tokens=0.812B | mem=4.2/32.7GB | cost=$0.76/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 00:46 (137.6h) | Phase3: SKIPPED | SFT: Oct 27 05:37 (142.4h) | DPO (Final): Oct 27 06:12 (143.0h) [2026-10-21 07:11:43][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step9 | Tokens: 0.812B | Spent: $0.76 | Remaining: $999.24 [2026-10-21 07:11:43][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step9 [2026-10-21 07:11:43][INFO][training.pretrain_engine] Starting global step 10 (accumulating 32 micro-batches)... [2026-10-21 07:11:43][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 07:11:43][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/agent-harnesses "HTTP/1.1 200 OK" [2026-10-21 07:11:43][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 07:11:43][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 07:11:43][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 07:11:43][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step9 to Phase-Technologies/forge-500m-pretrain (path: phase2_step9)... [2026-10-21 07:11:43][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 07:11:57][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 07:11:57][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step9 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 07:11:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6302 [2026-10-21 07:11:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6188 [2026-10-21 07:11:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4943 [2026-10-21 07:12:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6085 [2026-10-21 07:12:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3969 [2026-10-21 07:12:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1154 [2026-10-21 07:12:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3398 [2026-10-21 07:12:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0609 [2026-10-21 07:12:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4932 [2026-10-21 07:12:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3084 [2026-10-21 07:12:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1925 [2026-10-21 07:12:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6287 [2026-10-21 07:12:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2878 [2026-10-21 07:12:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4846 [2026-10-21 07:12:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4953 [2026-10-21 07:12:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3999 [2026-10-21 07:13:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0407 [2026-10-21 07:13:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6240 [2026-10-21 07:13:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2671 [2026-10-21 07:13:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1380 [2026-10-21 07:13:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2186 [2026-10-21 07:13:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0179 [2026-10-21 07:13:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7425 [2026-10-21 07:13:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5518 [2026-10-21 07:13:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7924 [2026-10-21 07:13:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9592 [2026-10-21 07:13:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4430 [2026-10-21 07:13:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3160 [2026-10-21 07:14:00][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,630 | MFU: 1.8% [2026-10-21 07:14:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5795 [2026-10-21 07:14:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3880 [2026-10-21 07:14:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7957 [2026-10-21 07:14:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5175 [2026-10-21 07:14:05][INFO][training.pretrain_engine] [Ph2|Step 10] loss=5.5592 | ppl=259.61 | aux=0.02509 | lr=3.00e-04 | gnorm=0.918 | tok/s=28,735 | MFU=6.9% | step=145965ms | tokens=0.813B | mem=4.2/32.7GB | cost=$0.84/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 00:22 (137.1h) | Phase3: SKIPPED | SFT: Oct 27 05:12 (142.0h) | DPO (Final): Oct 27 05:47 (142.6h) [2026-10-21 07:14:05][INFO][training.pretrain_engine] Starting global step 11 (accumulating 32 micro-batches)... [2026-10-21 07:14:05][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 07:14:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4247 [2026-10-21 07:14:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0939 [2026-10-21 07:14:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5028 [2026-10-21 07:14:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7974 [2026-10-21 07:14:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3401 [2026-10-21 07:14:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5070 [2026-10-21 07:14:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2672 [2026-10-21 07:14:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6912 [2026-10-21 07:14:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0656 [2026-10-21 07:14:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9139 [2026-10-21 07:14:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2206 [2026-10-21 07:14:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8683 [2026-10-21 07:15:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7871 [2026-10-21 07:15:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7621 [2026-10-21 07:15:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5980 [2026-10-21 07:15:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6615 [2026-10-21 07:15:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5195 [2026-10-21 07:15:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5520 [2026-10-21 07:15:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2656 [2026-10-21 07:15:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6400 [2026-10-21 07:15:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8208 [2026-10-21 07:15:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1433 [2026-10-21 07:15:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9509 [2026-10-21 07:15:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7117 [2026-10-21 07:16:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2290 [2026-10-21 07:16:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4684 [2026-10-21 07:16:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4602 [2026-10-21 07:16:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0088 [2026-10-21 07:16:22][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,623 | MFU: 1.8% [2026-10-21 07:16:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8106 [2026-10-21 07:16:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7197 [2026-10-21 07:16:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9132 [2026-10-21 07:16:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9285 [2026-10-21 07:16:27][INFO][training.pretrain_engine] [Ph2|Step 11] loss=5.4465 | ppl=231.95 | aux=0.02508 | lr=3.00e-04 | gnorm=0.973 | tok/s=28,794 | MFU=6.9% | step=145666ms | tokens=0.814B | mem=4.2/32.7GB | cost=$0.92/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 00:07 (136.9h) | Phase3: SKIPPED | SFT: Oct 27 04:57 (141.7h) | DPO (Final): Oct 27 05:32 (142.3h) [2026-10-21 07:16:27][INFO][training.pretrain_engine] Starting global step 12 (accumulating 32 micro-batches)... [2026-10-21 07:16:27][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 07:16:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2058 [2026-10-21 07:16:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5075 [2026-10-21 07:16:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1043 [2026-10-21 07:16:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5405 [2026-10-21 07:17:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4087 [2026-10-21 07:17:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1000 [2026-10-21 07:17:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6779 [2026-10-21 07:17:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2313 [2026-10-21 07:17:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5304 [2026-10-21 07:17:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5860 [2026-10-21 07:17:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4620 [2026-10-21 07:17:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0413 [2026-10-21 07:17:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5009 [2026-10-21 07:17:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2949 [2026-10-21 07:17:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2013 [2026-10-21 07:17:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2253 [2026-10-21 07:17:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3032 [2026-10-21 07:17:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7478 [2026-10-21 07:17:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0913 [2026-10-21 07:17:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5062 [2026-10-21 07:18:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5561 [2026-10-21 07:18:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2436 [2026-10-21 07:18:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8669 [2026-10-21 07:18:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2630 [2026-10-21 07:18:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2802 [2026-10-21 07:18:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0149 [2026-10-21 07:18:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3250 [2026-10-21 07:18:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8243 [2026-10-21 07:18:45][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,615 | MFU: 1.8% [2026-10-21 07:18:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4185 [2026-10-21 07:18:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0835 [2026-10-21 07:18:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8961 [2026-10-21 07:18:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1888 [2026-10-21 07:18:50][INFO][training.pretrain_engine] [Ph2|Step 12] loss=5.4153 | ppl=224.82 | aux=0.02508 | lr=3.00e-04 | gnorm=1.039 | tok/s=28,847 | MFU=6.9% | step=145398ms | tokens=0.815B | mem=4.2/32.7GB | cost=$1.01/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:54 (136.6h) | Phase3: SKIPPED | SFT: Oct 27 04:43 (141.4h) | DPO (Final): Oct 27 05:18 (142.0h) [2026-10-21 07:18:50][INFO][training.pretrain_engine] Starting global step 13 (accumulating 32 micro-batches)... [2026-10-21 07:18:50][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 07:19:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5289 [2026-10-21 07:19:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3024 [2026-10-21 07:19:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0696 [2026-10-21 07:19:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6332 [2026-10-21 07:19:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1386 [2026-10-21 07:19:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7841 [2026-10-21 07:19:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6311 [2026-10-21 07:19:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5513 [2026-10-21 07:19:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2405 [2026-10-21 07:19:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0992 [2026-10-21 07:19:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8191 [2026-10-21 07:19:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7952 [2026-10-21 07:20:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6915 [2026-10-21 07:20:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7587 [2026-10-21 07:20:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3878 [2026-10-21 07:20:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7367 [2026-10-21 07:20:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3636 [2026-10-21 07:20:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2668 [2026-10-21 07:20:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2488 [2026-10-21 07:20:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9916 [2026-10-21 07:20:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2432 [2026-10-21 07:20:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0342 [2026-10-21 07:20:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1292 [2026-10-21 07:20:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2676 [2026-10-21 07:20:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5032 [2026-10-21 07:20:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1246 [2026-10-21 07:20:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1706 [2026-10-21 07:20:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3392 [2026-10-21 07:21:07][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,640 | MFU: 1.8% [2026-10-21 07:21:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3025 [2026-10-21 07:21:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2578 [2026-10-21 07:21:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9050 [2026-10-21 07:21:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6307 [2026-10-21 07:21:12][INFO][training.pretrain_engine] [Ph2|Step 13] loss=5.3604 | ppl=212.81 | aux=0.02507 | lr=3.00e-04 | gnorm=0.941 | tok/s=28,890 | MFU=6.9% | step=145182ms | tokens=0.816B | mem=4.2/32.7GB | cost=$1.09/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:44 (136.4h) | Phase3: SKIPPED | SFT: Oct 27 04:32 (141.2h) | DPO (Final): Oct 27 05:07 (141.8h) [2026-10-21 07:21:12][INFO][training.pretrain_engine] Starting global step 14 (accumulating 32 micro-batches)... [2026-10-21 07:21:12][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 07:21:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5159 [2026-10-21 07:21:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3234 [2026-10-21 07:21:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5324 [2026-10-21 07:21:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4395 [2026-10-21 07:21:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6876 [2026-10-21 07:21:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5214 [2026-10-21 07:21:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5142 [2026-10-21 07:21:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6855 [2026-10-21 07:22:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5105 [2026-10-21 07:22:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5845 [2026-10-21 07:22:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4478 [2026-10-21 07:22:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3780 [2026-10-21 07:22:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0688 [2026-10-21 07:22:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9653 [2026-10-21 07:22:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8959 [2026-10-21 07:22:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8817 [2026-10-21 07:22:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9439 [2026-10-21 07:22:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8064 [2026-10-21 07:22:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7331 [2026-10-21 07:22:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5871 [2026-10-21 07:22:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5049 [2026-10-21 07:22:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0411 [2026-10-21 07:22:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4483 [2026-10-21 07:22:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2429 [2026-10-21 07:23:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7332 [2026-10-21 07:23:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6912 [2026-10-21 07:23:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4052 [2026-10-21 07:23:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1787 [2026-10-21 07:23:30][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,619 | MFU: 1.8% [2026-10-21 07:23:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4356 [2026-10-21 07:23:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6240 [2026-10-21 07:23:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2983 [2026-10-21 07:23:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3768 [2026-10-21 07:23:35][INFO][training.pretrain_engine] [Ph2|Step 14] loss=5.4279 | ppl=227.66 | aux=0.02507 | lr=3.00e-04 | gnorm=0.867 | tok/s=28,924 | MFU=6.9% | step=145010ms | tokens=0.817B | mem=4.2/32.7GB | cost=$1.17/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:36 (136.2h) | Phase3: SKIPPED | SFT: Oct 27 04:24 (141.0h) | DPO (Final): Oct 27 04:58 (141.6h) [2026-10-21 07:23:35][INFO][training.pretrain_engine] Starting global step 15 (accumulating 32 micro-batches)... [2026-10-21 07:23:35][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 07:23:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2636 [2026-10-21 07:23:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7392 [2026-10-21 07:23:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3364 [2026-10-21 07:23:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5840 [2026-10-21 07:24:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9355 [2026-10-21 07:24:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9325 [2026-10-21 07:24:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1707 [2026-10-21 07:24:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7876 [2026-10-21 07:24:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5569 [2026-10-21 07:24:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8372 [2026-10-21 07:24:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8630 [2026-10-21 07:24:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8342 [2026-10-21 07:24:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8396 [2026-10-21 07:24:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0785 [2026-10-21 07:24:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9946 [2026-10-21 07:24:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1212 [2026-10-21 07:25:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6066 [2026-10-21 07:25:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2388 [2026-10-21 07:25:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3212 [2026-10-21 07:25:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1840 [2026-10-21 07:25:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9751 [2026-10-21 07:25:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1875 [2026-10-21 07:25:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9340 [2026-10-21 07:25:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6091 [2026-10-21 07:25:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0357 [2026-10-21 07:25:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5291 [2026-10-21 07:25:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6157 [2026-10-21 07:25:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4802 [2026-10-21 07:25:53][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,627 | MFU: 1.8% [2026-10-21 07:25:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2001 [2026-10-21 07:25:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5585 [2026-10-21 07:25:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5521 [2026-10-21 07:25:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0572 [2026-10-21 07:25:59][INFO][training.pretrain_engine] [Ph2|Step 15] loss=5.3701 | ppl=214.89 | aux=0.02508 | lr=3.00e-04 | gnorm=1.039 | tok/s=28,946 | MFU=6.9% | step=144902ms | tokens=0.818B | mem=4.2/32.7GB | cost=$1.25/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:31 (136.1h) | Phase3: SKIPPED | SFT: Oct 27 04:19 (140.9h) | DPO (Final): Oct 27 04:54 (141.5h) [2026-10-21 07:25:59][INFO][training.pretrain_engine] Starting global step 16 (accumulating 32 micro-batches)... [2026-10-21 07:25:59][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 07:26:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6424 [2026-10-21 07:26:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3000 [2026-10-21 07:26:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0172 [2026-10-21 07:26:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0356 [2026-10-21 07:26:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3898 [2026-10-21 07:26:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2667 [2026-10-21 07:26:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3947 [2026-10-21 07:26:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5193 [2026-10-21 07:26:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8444 [2026-10-21 07:26:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4487 [2026-10-21 07:26:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9856 [2026-10-21 07:26:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8177 [2026-10-21 07:27:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0528 [2026-10-21 07:27:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2085 [2026-10-21 07:27:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5037 [2026-10-21 07:27:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1087 [2026-10-21 07:27:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0704 [2026-10-21 07:27:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0130 [2026-10-21 07:27:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0649 [2026-10-21 07:27:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5907 [2026-10-21 07:27:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3344 [2026-10-21 07:27:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5290 [2026-10-21 07:27:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0312 [2026-10-21 07:27:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2130 [2026-10-21 07:28:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3391 [2026-10-21 07:28:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5989 [2026-10-21 07:28:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4056 [2026-10-21 07:28:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5055 [2026-10-21 07:28:16][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,625 | MFU: 1.8% [2026-10-21 07:28:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4597 [2026-10-21 07:28:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9962 [2026-10-21 07:28:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1239 [2026-10-21 07:28:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0255 [2026-10-21 07:28:21][INFO][training.pretrain_engine] [Ph2|Step 16] loss=5.5336 | ppl=253.04 | aux=0.02508 | lr=3.00e-04 | gnorm=1.094 | tok/s=28,974 | MFU=7.0% | step=144763ms | tokens=0.819B | mem=4.2/32.7GB | cost=$1.33/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:25 (136.0h) | Phase3: SKIPPED | SFT: Oct 27 04:13 (140.8h) | DPO (Final): Oct 27 04:47 (141.3h) [2026-10-21 07:28:21][INFO][training.pretrain_engine] Starting global step 17 (accumulating 32 micro-batches)... [2026-10-21 07:28:21][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 07:28:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5499 [2026-10-21 07:28:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7086 [2026-10-21 07:28:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0910 [2026-10-21 07:28:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6471 [2026-10-21 07:28:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5305 [2026-10-21 07:28:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6540 [2026-10-21 07:28:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8856 [2026-10-21 07:28:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0715 [2026-10-21 07:29:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6557 [2026-10-21 07:29:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5425 [2026-10-21 07:29:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6155 [2026-10-21 07:29:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5825 [2026-10-21 07:29:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5184 [2026-10-21 07:29:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6244 [2026-10-21 07:29:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9008 [2026-10-21 07:29:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4359 [2026-10-21 07:29:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5486 [2026-10-21 07:29:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8971 [2026-10-21 07:29:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6364 [2026-10-21 07:29:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6012 [2026-10-21 07:30:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5325 [2026-10-21 07:30:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2402 [2026-10-21 07:30:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4070 [2026-10-21 07:30:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4439 [2026-10-21 07:30:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1311 [2026-10-21 07:30:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9041 [2026-10-21 07:30:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1646 [2026-10-21 07:30:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5469 [2026-10-21 07:30:39][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,616 | MFU: 1.8% [2026-10-21 07:30:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6680 [2026-10-21 07:30:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8537 [2026-10-21 07:30:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4498 [2026-10-21 07:30:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3913 [2026-10-21 07:30:44][INFO][training.pretrain_engine] [Ph2|Step 17] loss=5.3896 | ppl=219.11 | aux=0.02508 | lr=3.00e-04 | gnorm=0.781 | tok/s=28,995 | MFU=7.0% | step=144656ms | tokens=0.820B | mem=4.2/32.7GB | cost=$1.42/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:21 (135.8h) | Phase3: SKIPPED | SFT: Oct 27 04:08 (140.6h) | DPO (Final): Oct 27 04:43 (141.2h) [2026-10-21 07:30:44][INFO][training.pretrain_engine] Starting global step 18 (accumulating 32 micro-batches)... [2026-10-21 07:30:44][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 07:31:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4295 [2026-10-21 07:31:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6161 [2026-10-21 07:31:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5683 [2026-10-21 07:31:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9299 [2026-10-21 07:31:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6251 [2026-10-21 07:31:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8947 [2026-10-21 07:31:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.8048 [2026-10-21 07:31:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9440 [2026-10-21 07:31:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4025 [2026-10-21 07:31:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1512 [2026-10-21 07:31:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4555 [2026-10-21 07:31:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5565 [2026-10-21 07:31:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6870 [2026-10-21 07:31:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5389 [2026-10-21 07:31:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3387 [2026-10-21 07:31:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6510 [2026-10-21 07:32:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1382 [2026-10-21 07:32:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2014 [2026-10-21 07:32:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0855 [2026-10-21 07:32:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1275 [2026-10-21 07:32:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=3.6350 [2026-10-21 07:32:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9691 [2026-10-21 07:32:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5083 [2026-10-21 07:32:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7096 [2026-10-21 07:32:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4044 [2026-10-21 07:32:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3349 [2026-10-21 07:32:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5721 [2026-10-21 07:32:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9684 [2026-10-21 07:33:02][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,616 | MFU: 1.8% [2026-10-21 07:33:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1795 [2026-10-21 07:33:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3207 [2026-10-21 07:33:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3527 [2026-10-21 07:33:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2764 [2026-10-21 07:33:07][INFO][training.pretrain_engine] [Ph2|Step 18] loss=5.3786 | ppl=216.71 | aux=0.02507 | lr=3.00e-04 | gnorm=0.594 | tok/s=29,014 | MFU=7.0% | step=144563ms | tokens=0.821B | mem=4.2/32.7GB | cost=$1.50/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:18 (135.7h) | Phase3: SKIPPED | SFT: Oct 27 04:05 (140.5h) | DPO (Final): Oct 27 04:39 (141.1h) [2026-10-21 07:33:10][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step18 | Tokens: 0.821B | Spent: $1.50 | Remaining: $998.50 [2026-10-21 07:33:10][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step18 [2026-10-21 07:33:10][INFO][training.pretrain_engine] Starting global step 19 (accumulating 32 micro-batches)... [2026-10-21 07:33:10][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 07:33:10][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 07:33:10][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 07:33:10][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 07:33:10][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step18 to Phase-Technologies/forge-500m-pretrain (path: phase2_step18)... [2026-10-21 07:33:11][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 07:33:23][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 07:33:23][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step18 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 07:33:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9285 [2026-10-21 07:33:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2360 [2026-10-21 07:33:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8723 [2026-10-21 07:33:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0743 [2026-10-21 07:33:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4876 [2026-10-21 07:33:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6724 [2026-10-21 07:33:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2955 [2026-10-21 07:33:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9372 [2026-10-21 07:34:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8802 [2026-10-21 07:34:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5510 [2026-10-21 07:34:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5064 [2026-10-21 07:34:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6966 [2026-10-21 07:34:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7433 [2026-10-21 07:34:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5720 [2026-10-21 07:34:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5517 [2026-10-21 07:34:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5744 [2026-10-21 07:34:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4458 [2026-10-21 07:34:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.5498 [2026-10-21 07:34:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6133 [2026-10-21 07:34:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5092 [2026-10-21 07:34:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9732 [2026-10-21 07:34:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1638 [2026-10-21 07:34:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3337 [2026-10-21 07:34:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5085 [2026-10-21 07:35:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9744 [2026-10-21 07:35:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5469 [2026-10-21 07:35:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6474 [2026-10-21 07:35:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7559 [2026-10-21 07:35:28][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,622 | MFU: 1.8% [2026-10-21 07:35:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2315 [2026-10-21 07:35:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3297 [2026-10-21 07:35:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4321 [2026-10-21 07:35:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2690 [2026-10-21 07:35:32][INFO][training.pretrain_engine] [Ph2|Step 19] loss=5.4872 | ppl=241.59 | aux=0.02507 | lr=3.00e-04 | gnorm=0.840 | tok/s=29,041 | MFU=7.0% | step=144429ms | tokens=0.822B | mem=4.2/32.7GB | cost=$1.58/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:12 (135.6h) | Phase3: SKIPPED | SFT: Oct 27 03:59 (140.4h) | DPO (Final): Oct 27 04:33 (141.0h) [2026-10-21 07:35:32][INFO][training.pretrain_engine] Starting global step 20 (accumulating 32 micro-batches)... [2026-10-21 07:35:32][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 07:35:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2855 [2026-10-21 07:35:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5748 [2026-10-21 07:35:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2459 [2026-10-21 07:35:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1066 [2026-10-21 07:36:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2608 [2026-10-21 07:36:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6647 [2026-10-21 07:36:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8632 [2026-10-21 07:36:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5350 [2026-10-21 07:36:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4837 [2026-10-21 07:36:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7345 [2026-10-21 07:36:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2090 [2026-10-21 07:36:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4684 [2026-10-21 07:36:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1193 [2026-10-21 07:36:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2624 [2026-10-21 07:36:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3817 [2026-10-21 07:36:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2455 [2026-10-21 07:37:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9005 [2026-10-21 07:37:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5523 [2026-10-21 07:37:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7002 [2026-10-21 07:37:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7552 [2026-10-21 07:37:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6039 [2026-10-21 07:37:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9263 [2026-10-21 07:37:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2997 [2026-10-21 07:37:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3298 [2026-10-21 07:37:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5682 [2026-10-21 07:37:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2680 [2026-10-21 07:37:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2188 [2026-10-21 07:37:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1066 [2026-10-21 07:37:50][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,632 | MFU: 1.8% [2026-10-21 07:37:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0908 [2026-10-21 07:37:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5351 [2026-10-21 07:37:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2676 [2026-10-21 07:37:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4216 [2026-10-21 07:37:55][INFO][training.pretrain_engine] [Ph2|Step 20] loss=5.4431 | ppl=231.17 | aux=0.02507 | lr=3.00e-04 | gnorm=0.777 | tok/s=29,057 | MFU=7.0% | step=144349ms | tokens=0.823B | mem=4.2/32.7GB | cost=$1.66/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:09 (135.5h) | Phase3: SKIPPED | SFT: Oct 27 03:56 (140.3h) | DPO (Final): Oct 27 04:30 (140.9h) [2026-10-21 07:37:55][INFO][training.pretrain_engine] Starting global step 21 (accumulating 32 micro-batches)... [2026-10-21 07:37:55][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 07:38:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9110 [2026-10-21 07:38:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9480 [2026-10-21 07:38:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6843 [2026-10-21 07:38:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7452 [2026-10-21 07:38:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4251 [2026-10-21 07:38:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0425 [2026-10-21 07:38:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4200 [2026-10-21 07:38:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9855 [2026-10-21 07:38:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5680 [2026-10-21 07:38:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4238 [2026-10-21 07:38:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6377 [2026-10-21 07:38:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4659 [2026-10-21 07:39:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0587 [2026-10-21 07:39:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1057 [2026-10-21 07:39:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8569 [2026-10-21 07:39:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1698 [2026-10-21 07:39:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4364 [2026-10-21 07:39:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2859 [2026-10-21 07:39:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4700 [2026-10-21 07:39:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6069 [2026-10-21 07:39:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0153 [2026-10-21 07:39:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5854 [2026-10-21 07:39:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3194 [2026-10-21 07:39:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1788 [2026-10-21 07:39:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1102 [2026-10-21 07:39:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8391 [2026-10-21 07:40:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7595 [2026-10-21 07:40:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0152 [2026-10-21 07:40:12][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,633 | MFU: 1.8% [2026-10-21 07:40:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2651 [2026-10-21 07:40:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9358 [2026-10-21 07:40:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7995 [2026-10-21 07:40:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2403 [2026-10-21 07:40:18][INFO][training.pretrain_engine] [Ph2|Step 21] loss=5.2972 | ppl=199.78 | aux=0.02507 | lr=3.00e-04 | gnorm=0.660 | tok/s=29,074 | MFU=7.0% | step=144264ms | tokens=0.824B | mem=4.2/32.7GB | cost=$1.74/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:06 (135.4h) | Phase3: SKIPPED | SFT: Oct 27 03:53 (140.2h) | DPO (Final): Oct 27 04:27 (140.8h) [2026-10-21 07:40:18][INFO][training.pretrain_engine] Starting global step 22 (accumulating 32 micro-batches)... [2026-10-21 07:40:18][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 07:40:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1968 [2026-10-21 07:40:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2764 [2026-10-21 07:40:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0198 [2026-10-21 07:40:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0089 [2026-10-21 07:40:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3246 [2026-10-21 07:40:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3839 [2026-10-21 07:40:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5634 [2026-10-21 07:40:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0213 [2026-10-21 07:41:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2296 [2026-10-21 07:41:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3092 [2026-10-21 07:41:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3203 [2026-10-21 07:41:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2831 [2026-10-21 07:41:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0838 [2026-10-21 07:41:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9486 [2026-10-21 07:41:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1739 [2026-10-21 07:41:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1982 [2026-10-21 07:41:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1195 [2026-10-21 07:41:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2986 [2026-10-21 07:41:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1789 [2026-10-21 07:41:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0091 [2026-10-21 07:42:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9676 [2026-10-21 07:42:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1995 [2026-10-21 07:42:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5470 [2026-10-21 07:42:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6907 [2026-10-21 07:42:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9869 [2026-10-21 07:42:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2808 [2026-10-21 07:42:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4254 [2026-10-21 07:42:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7010 [2026-10-21 07:42:35][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,608 | MFU: 1.8% [2026-10-21 07:42:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6791 [2026-10-21 07:42:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3641 [2026-10-21 07:42:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5393 [2026-10-21 07:42:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4818 [2026-10-21 07:42:41][INFO][training.pretrain_engine] [Ph2|Step 22] loss=5.3690 | ppl=214.65 | aux=0.02507 | lr=3.00e-04 | gnorm=0.633 | tok/s=29,083 | MFU=7.0% | step=144216ms | tokens=0.825B | mem=4.2/32.7GB | cost=$1.83/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:05 (135.4h) | Phase3: SKIPPED | SFT: Oct 27 03:52 (140.2h) | DPO (Final): Oct 27 04:26 (140.7h) [2026-10-21 07:42:41][INFO][training.pretrain_engine] Starting global step 23 (accumulating 32 micro-batches)... [2026-10-21 07:42:41][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 07:42:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6880 [2026-10-21 07:42:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3628 [2026-10-21 07:42:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8138 [2026-10-21 07:42:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6860 [2026-10-21 07:43:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7516 [2026-10-21 07:43:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4439 [2026-10-21 07:43:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3309 [2026-10-21 07:43:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5437 [2026-10-21 07:43:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4508 [2026-10-21 07:43:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1989 [2026-10-21 07:43:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2752 [2026-10-21 07:43:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2830 [2026-10-21 07:43:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6938 [2026-10-21 07:43:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2040 [2026-10-21 07:43:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9974 [2026-10-21 07:43:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2092 [2026-10-21 07:44:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7021 [2026-10-21 07:44:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6714 [2026-10-21 07:44:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7813 [2026-10-21 07:44:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7399 [2026-10-21 07:44:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9898 [2026-10-21 07:44:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7756 [2026-10-21 07:44:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2279 [2026-10-21 07:44:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6886 [2026-10-21 07:44:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5749 [2026-10-21 07:44:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9648 [2026-10-21 07:44:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5439 [2026-10-21 07:44:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6346 [2026-10-21 07:44:58][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,629 | MFU: 1.8% [2026-10-21 07:45:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3801 [2026-10-21 07:45:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2786 [2026-10-21 07:45:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1716 [2026-10-21 07:45:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5288 [2026-10-21 07:45:04][INFO][training.pretrain_engine] [Ph2|Step 23] loss=5.3731 | ppl=215.53 | aux=0.02507 | lr=3.00e-04 | gnorm=0.641 | tok/s=29,092 | MFU=7.0% | step=144173ms | tokens=0.826B | mem=4.2/32.7GB | cost=$1.91/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:05 (135.3h) | Phase3: SKIPPED | SFT: Oct 27 03:51 (140.1h) | DPO (Final): Oct 27 04:25 (140.7h) [2026-10-21 07:45:04][INFO][training.pretrain_engine] Starting global step 24 (accumulating 32 micro-batches)... [2026-10-21 07:45:04][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 07:45:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2888 [2026-10-21 07:45:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2666 [2026-10-21 07:45:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6213 [2026-10-21 07:45:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4512 [2026-10-21 07:45:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3479 [2026-10-21 07:45:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5192 [2026-10-21 07:45:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8347 [2026-10-21 07:45:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2416 [2026-10-21 07:45:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2444 [2026-10-21 07:45:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1455 [2026-10-21 07:45:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7429 [2026-10-21 07:45:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1322 [2026-10-21 07:46:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2959 [2026-10-21 07:46:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3236 [2026-10-21 07:46:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9643 [2026-10-21 07:46:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9665 [2026-10-21 07:46:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8010 [2026-10-21 07:46:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0588 [2026-10-21 07:46:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0944 [2026-10-21 07:46:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8011 [2026-10-21 07:46:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4016 [2026-10-21 07:46:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0949 [2026-10-21 07:46:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1088 [2026-10-21 07:46:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7628 [2026-10-21 07:47:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0910 [2026-10-21 07:47:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5172 [2026-10-21 07:47:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5741 [2026-10-21 07:47:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5379 [2026-10-21 07:47:21][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,638 | MFU: 1.8% [2026-10-21 07:47:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5532 [2026-10-21 07:47:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5513 [2026-10-21 07:47:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5003 [2026-10-21 07:47:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3766 [2026-10-21 07:47:27][INFO][training.pretrain_engine] [Ph2|Step 24] loss=5.3696 | ppl=214.78 | aux=0.02507 | lr=3.00e-04 | gnorm=0.660 | tok/s=29,103 | MFU=7.0% | step=144120ms | tokens=0.827B | mem=4.2/32.7GB | cost=$1.99/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:03 (135.3h) | Phase3: SKIPPED | SFT: Oct 27 03:50 (140.0h) | DPO (Final): Oct 27 04:24 (140.6h) [2026-10-21 07:47:27][INFO][training.pretrain_engine] Starting global step 25 (accumulating 32 micro-batches)... [2026-10-21 07:47:27][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 07:47:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5527 [2026-10-21 07:47:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5873 [2026-10-21 07:47:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8910 [2026-10-21 07:47:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0696 [2026-10-21 07:48:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3672 [2026-10-21 07:48:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2205 [2026-10-21 07:48:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5727 [2026-10-21 07:48:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1883 [2026-10-21 07:48:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6949 [2026-10-21 07:48:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5191 [2026-10-21 07:48:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4803 [2026-10-21 07:48:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4020 [2026-10-21 07:48:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2945 [2026-10-21 07:48:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3572 [2026-10-21 07:48:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4074 [2026-10-21 07:48:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8725 [2026-10-21 07:48:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3947 [2026-10-21 07:48:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8206 [2026-10-21 07:48:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4330 [2026-10-21 07:48:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2065 [2026-10-21 07:49:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7392 [2026-10-21 07:49:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6339 [2026-10-21 07:49:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3576 [2026-10-21 07:49:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5418 [2026-10-21 07:49:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2915 [2026-10-21 07:49:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9347 [2026-10-21 07:49:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3803 [2026-10-21 07:49:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9440 [2026-10-21 07:49:45][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,614 | MFU: 1.8% [2026-10-21 07:49:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4725 [2026-10-21 07:49:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5501 [2026-10-21 07:49:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0666 [2026-10-21 07:49:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3373 [2026-10-21 07:49:50][INFO][training.pretrain_engine] [Ph2|Step 25] loss=5.4473 | ppl=232.13 | aux=0.02507 | lr=3.00e-04 | gnorm=0.566 | tok/s=29,112 | MFU=7.0% | step=144073ms | tokens=0.828B | mem=4.2/32.7GB | cost=$2.07/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:03 (135.2h) | Phase3: SKIPPED | SFT: Oct 27 03:49 (140.0h) | DPO (Final): Oct 27 04:23 (140.6h) [2026-10-21 07:49:50][INFO][training.pretrain_engine] Starting global step 26 (accumulating 32 micro-batches)... [2026-10-21 07:49:50][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 07:50:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9149 [2026-10-21 07:50:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3797 [2026-10-21 07:50:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2769 [2026-10-21 07:50:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9794 [2026-10-21 07:50:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1900 [2026-10-21 07:50:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8669 [2026-10-21 07:50:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8717 [2026-10-21 07:50:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8939 [2026-10-21 07:50:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6587 [2026-10-21 07:50:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9189 [2026-10-21 07:50:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8815 [2026-10-21 07:50:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4737 [2026-10-21 07:51:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4171 [2026-10-21 07:51:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3988 [2026-10-21 07:51:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3736 [2026-10-21 07:51:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4025 [2026-10-21 07:51:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5279 [2026-10-21 07:51:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2510 [2026-10-21 07:51:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3696 [2026-10-21 07:51:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3480 [2026-10-21 07:51:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3898 [2026-10-21 07:51:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6576 [2026-10-21 07:51:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3975 [2026-10-21 07:51:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6295 [2026-10-21 07:51:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2872 [2026-10-21 07:51:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8603 [2026-10-21 07:51:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9183 [2026-10-21 07:51:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7615 [2026-10-21 07:52:07][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,635 | MFU: 1.8% [2026-10-21 07:52:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2623 [2026-10-21 07:52:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6177 [2026-10-21 07:52:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5991 [2026-10-21 07:52:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3409 [2026-10-21 07:52:13][INFO][training.pretrain_engine] [Ph2|Step 26] loss=5.4456 | ppl=231.74 | aux=0.02508 | lr=3.00e-04 | gnorm=0.566 | tok/s=29,119 | MFU=7.0% | step=144041ms | tokens=0.829B | mem=4.2/32.7GB | cost=$2.15/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:03 (135.2h) | Phase3: SKIPPED | SFT: Oct 27 03:49 (139.9h) | DPO (Final): Oct 27 04:23 (140.5h) [2026-10-21 07:52:13][INFO][training.pretrain_engine] Starting global step 27 (accumulating 32 micro-batches)... [2026-10-21 07:52:13][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 07:52:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3945 [2026-10-21 07:52:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4450 [2026-10-21 07:52:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1095 [2026-10-21 07:52:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3635 [2026-10-21 07:52:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6690 [2026-10-21 07:52:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5082 [2026-10-21 07:52:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5706 [2026-10-21 07:52:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6582 [2026-10-21 07:53:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4631 [2026-10-21 07:53:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7420 [2026-10-21 07:53:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7667 [2026-10-21 07:53:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8343 [2026-10-21 07:53:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0386 [2026-10-21 07:53:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0747 [2026-10-21 07:53:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2206 [2026-10-21 07:53:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0657 [2026-10-21 07:53:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2008 [2026-10-21 07:53:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5343 [2026-10-21 07:53:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4918 [2026-10-21 07:53:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5487 [2026-10-21 07:53:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2954 [2026-10-21 07:54:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6732 [2026-10-21 07:54:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3543 [2026-10-21 07:54:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0376 [2026-10-21 07:54:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2558 [2026-10-21 07:54:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6211 [2026-10-21 07:54:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7338 [2026-10-21 07:54:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1935 [2026-10-21 07:54:31][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,623 | MFU: 1.8% [2026-10-21 07:54:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6345 [2026-10-21 07:54:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8784 [2026-10-21 07:54:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9461 [2026-10-21 07:54:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3896 [2026-10-21 07:54:37][INFO][training.pretrain_engine] [Ph2|Step 27] loss=5.4304 | ppl=228.25 | aux=0.02507 | lr=3.00e-04 | gnorm=0.562 | tok/s=29,123 | MFU=7.0% | step=144020ms | tokens=0.830B | mem=4.2/32.7GB | cost=$2.23/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:03 (135.1h) | Phase3: SKIPPED | SFT: Oct 27 03:49 (139.9h) | DPO (Final): Oct 27 04:24 (140.5h) [2026-10-21 07:54:40][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step27 | Tokens: 0.830B | Spent: $2.24 | Remaining: $997.76 [2026-10-21 07:54:40][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step27 [2026-10-21 07:54:40][INFO][training.pretrain_engine] Starting global step 28 (accumulating 32 micro-batches)... [2026-10-21 07:54:40][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 07:54:40][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 07:54:40][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 07:54:40][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 07:54:40][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step27 to Phase-Technologies/forge-500m-pretrain (path: phase2_step27)... [2026-10-21 07:54:40][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 07:54:52][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 07:54:52][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step27 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 07:54:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6816 [2026-10-21 07:54:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5612 [2026-10-21 07:54:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4442 [2026-10-21 07:54:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3885 [2026-10-21 07:55:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8533 [2026-10-21 07:55:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3654 [2026-10-21 07:55:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3484 [2026-10-21 07:55:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9552 [2026-10-21 07:55:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8622 [2026-10-21 07:55:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3138 [2026-10-21 07:55:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7995 [2026-10-21 07:55:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8507 [2026-10-21 07:55:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0256 [2026-10-21 07:55:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4016 [2026-10-21 07:55:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2939 [2026-10-21 07:55:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3263 [2026-10-21 07:56:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0721 [2026-10-21 07:56:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8362 [2026-10-21 07:56:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9412 [2026-10-21 07:56:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8208 [2026-10-21 07:56:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5977 [2026-10-21 07:56:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4661 [2026-10-21 07:56:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7964 [2026-10-21 07:56:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2730 [2026-10-21 07:56:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4940 [2026-10-21 07:56:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.7119 [2026-10-21 07:56:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5613 [2026-10-21 07:56:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7112 [2026-10-21 07:56:57][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,622 | MFU: 1.8% [2026-10-21 07:57:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1223 [2026-10-21 07:57:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5489 [2026-10-21 07:57:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6223 [2026-10-21 07:57:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0444 [2026-10-21 07:57:02][INFO][training.pretrain_engine] [Ph2|Step 28] loss=5.3861 | ppl=218.35 | aux=0.02510 | lr=3.00e-04 | gnorm=0.590 | tok/s=29,137 | MFU=7.0% | step=143949ms | tokens=0.832B | mem=4.2/32.7GB | cost=$2.32/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:01 (135.1h) | Phase3: SKIPPED | SFT: Oct 27 03:47 (139.8h) | DPO (Final): Oct 27 04:21 (140.4h) [2026-10-21 07:57:02][INFO][training.pretrain_engine] Starting global step 29 (accumulating 32 micro-batches)... [2026-10-21 07:57:02][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 07:57:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9659 [2026-10-21 07:57:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1327 [2026-10-21 07:57:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5988 [2026-10-21 07:57:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3528 [2026-10-21 07:57:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9840 [2026-10-21 07:57:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3962 [2026-10-21 07:57:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6191 [2026-10-21 07:57:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9728 [2026-10-21 07:57:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1961 [2026-10-21 07:57:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5089 [2026-10-21 07:57:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2545 [2026-10-21 07:57:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2465 [2026-10-21 07:58:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5791 [2026-10-21 07:58:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4539 [2026-10-21 07:58:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2055 [2026-10-21 07:58:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9062 [2026-10-21 07:58:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5433 [2026-10-21 07:58:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7740 [2026-10-21 07:58:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8961 [2026-10-21 07:58:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6009 [2026-10-21 07:58:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0418 [2026-10-21 07:58:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2759 [2026-10-21 07:58:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7558 [2026-10-21 07:58:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6794 [2026-10-21 07:59:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2187 [2026-10-21 07:59:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1500 [2026-10-21 07:59:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3646 [2026-10-21 07:59:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1966 [2026-10-21 07:59:19][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,617 | MFU: 1.8% [2026-10-21 07:59:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4641 [2026-10-21 07:59:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8014 [2026-10-21 07:59:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9362 [2026-10-21 07:59:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8198 [2026-10-21 07:59:24][INFO][training.pretrain_engine] [Ph2|Step 29] loss=5.3664 | ppl=214.10 | aux=0.02507 | lr=3.00e-04 | gnorm=0.574 | tok/s=29,148 | MFU=7.0% | step=143895ms | tokens=0.833B | mem=4.2/32.7GB | cost=$2.40/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:00 (135.0h) | Phase3: SKIPPED | SFT: Oct 27 03:46 (139.8h) | DPO (Final): Oct 27 04:20 (140.3h) [2026-10-21 07:59:24][INFO][training.pretrain_engine] Starting global step 30 (accumulating 32 micro-batches)... [2026-10-21 07:59:24][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 07:59:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5368 [2026-10-21 07:59:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2064 [2026-10-21 07:59:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4447 [2026-10-21 07:59:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2137 [2026-10-21 07:59:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5320 [2026-10-21 07:59:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8119 [2026-10-21 07:59:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8784 [2026-10-21 08:00:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0887 [2026-10-21 08:00:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2754 [2026-10-21 08:00:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2720 [2026-10-21 08:00:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4695 [2026-10-21 08:00:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6483 [2026-10-21 08:00:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5146 [2026-10-21 08:00:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3373 [2026-10-21 08:00:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4239 [2026-10-21 08:00:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7515 [2026-10-21 08:00:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6849 [2026-10-21 08:00:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9546 [2026-10-21 08:00:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.2702 [2026-10-21 08:00:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6478 [2026-10-21 08:01:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5947 [2026-10-21 08:01:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8597 [2026-10-21 08:01:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6454 [2026-10-21 08:01:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4569 [2026-10-21 08:01:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4171 [2026-10-21 08:01:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4297 [2026-10-21 08:01:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4648 [2026-10-21 08:01:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1087 [2026-10-21 08:01:42][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,609 | MFU: 1.8% [2026-10-21 08:01:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5753 [2026-10-21 08:01:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3793 [2026-10-21 08:01:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7102 [2026-10-21 08:01:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8864 [2026-10-21 08:01:46][INFO][training.pretrain_engine] [Ph2|Step 30] loss=5.4091 | ppl=223.43 | aux=0.02507 | lr=3.00e-04 | gnorm=0.543 | tok/s=29,158 | MFU=7.0% | step=143848ms | tokens=0.834B | mem=4.2/32.7GB | cost=$2.48/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 22:59 (135.0h) | Phase3: SKIPPED | SFT: Oct 27 03:45 (139.7h) | DPO (Final): Oct 27 04:19 (140.3h) [2026-10-21 08:01:46][INFO][training.pretrain_engine] Starting global step 31 (accumulating 32 micro-batches)... [2026-10-21 08:01:46][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 08:02:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4083 [2026-10-21 08:02:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3591 [2026-10-21 08:02:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3796 [2026-10-21 08:02:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8106 [2026-10-21 08:02:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4989 [2026-10-21 08:02:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6741 [2026-10-21 08:02:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5795 [2026-10-21 08:02:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5485 [2026-10-21 08:02:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4981 [2026-10-21 08:02:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7606 [2026-10-21 08:02:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8837 [2026-10-21 08:02:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6626 [2026-10-21 08:02:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7943 [2026-10-21 08:02:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3798 [2026-10-21 08:02:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1176 [2026-10-21 08:02:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2286 [2026-10-21 08:03:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3210 [2026-10-21 08:03:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1473 [2026-10-21 08:03:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9545 [2026-10-21 08:03:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2859 [2026-10-21 08:03:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0331 [2026-10-21 08:03:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2429 [2026-10-21 08:03:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1069 [2026-10-21 08:03:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1978 [2026-10-21 08:03:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1327 [2026-10-21 08:03:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3931 [2026-10-21 08:03:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3910 [2026-10-21 08:03:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3092 [2026-10-21 08:04:04][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,647 | MFU: 1.8% [2026-10-21 08:04:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7674 [2026-10-21 08:04:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3062 [2026-10-21 08:04:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6316 [2026-10-21 08:04:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7000 [2026-10-21 08:04:09][INFO][training.pretrain_engine] [Ph2|Step 31] loss=5.3761 | ppl=216.17 | aux=0.02508 | lr=3.00e-04 | gnorm=0.527 | tok/s=29,166 | MFU=7.0% | step=143809ms | tokens=0.835B | mem=4.2/32.7GB | cost=$2.56/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 22:58 (134.9h) | Phase3: SKIPPED | SFT: Oct 27 03:44 (139.7h) | DPO (Final): Oct 27 04:18 (140.2h) [2026-10-21 08:04:09][INFO][training.pretrain_engine] Starting global step 32 (accumulating 32 micro-batches)... [2026-10-21 08:04:09][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 08:04:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3893 [2026-10-21 08:04:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8826 [2026-10-21 08:04:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6689 [2026-10-21 08:04:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9930 [2026-10-21 08:04:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5415 [2026-10-21 08:04:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0207 [2026-10-21 08:04:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5554 [2026-10-21 08:04:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4522 [2026-10-21 08:05:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6678 [2026-10-21 08:05:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3398 [2026-10-21 08:05:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6883 [2026-10-21 08:05:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4481 [2026-10-21 08:05:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6995 [2026-10-21 08:05:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9829 [2026-10-21 08:05:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6054 [2026-10-21 08:05:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6580 [2026-10-21 08:05:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4150 [2026-10-21 08:05:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3736 [2026-10-21 08:05:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6313 [2026-10-21 08:05:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4118 [2026-10-21 08:05:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0837 [2026-10-21 08:05:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5374 [2026-10-21 08:05:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5562 [2026-10-21 08:05:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5328 [2026-10-21 08:06:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6744 [2026-10-21 08:06:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1672 [2026-10-21 08:06:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1932 [2026-10-21 08:06:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3828 [2026-10-21 08:06:27][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,622 | MFU: 1.8% [2026-10-21 08:06:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4320 [2026-10-21 08:06:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3904 [2026-10-21 08:06:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4038 [2026-10-21 08:06:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2913 [2026-10-21 08:06:31][INFO][training.pretrain_engine] [Ph2|Step 32] loss=5.4193 | ppl=225.72 | aux=0.02508 | lr=3.00e-04 | gnorm=0.535 | tok/s=29,175 | MFU=7.0% | step=143764ms | tokens=0.836B | mem=4.2/32.7GB | cost=$2.64/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 22:58 (134.9h) | Phase3: SKIPPED | SFT: Oct 27 03:43 (139.6h) | DPO (Final): Oct 27 04:18 (140.2h) [2026-10-21 08:06:31][INFO][training.pretrain_engine] Starting global step 33 (accumulating 32 micro-batches)... [2026-10-21 08:06:31][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 08:06:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4341 [2026-10-21 08:06:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2791 [2026-10-21 08:06:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5294 [2026-10-21 08:06:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2398 [2026-10-21 08:07:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0584 [2026-10-21 08:07:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1770 [2026-10-21 08:07:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5156 [2026-10-21 08:07:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1494 [2026-10-21 08:07:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9752 [2026-10-21 08:07:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7123 [2026-10-21 08:07:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0098 [2026-10-21 08:07:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7884 [2026-10-21 08:07:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5224 [2026-10-21 08:07:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5302 [2026-10-21 08:07:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0563 [2026-10-21 08:07:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3652 [2026-10-21 08:08:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4424 [2026-10-21 08:08:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6063 [2026-10-21 08:08:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3005 [2026-10-21 08:08:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2004 [2026-10-21 08:08:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2197 [2026-10-21 08:08:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7389 [2026-10-21 08:08:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5756 [2026-10-21 08:08:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9024 [2026-10-21 08:08:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5728 [2026-10-21 08:08:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3433 [2026-10-21 08:08:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2817 [2026-10-21 08:08:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3645 [2026-10-21 08:08:49][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,636 | MFU: 1.8% [2026-10-21 08:08:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5745 [2026-10-21 08:08:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6490 [2026-10-21 08:08:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4561 [2026-10-21 08:08:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=3.7918 [2026-10-21 08:08:54][INFO][training.pretrain_engine] [Ph2|Step 33] loss=5.2597 | ppl=192.43 | aux=0.02507 | lr=3.00e-04 | gnorm=0.551 | tok/s=29,181 | MFU=7.0% | step=143734ms | tokens=0.837B | mem=4.2/32.7GB | cost=$2.73/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 22:58 (134.8h) | Phase3: SKIPPED | SFT: Oct 27 03:43 (139.6h) | DPO (Final): Oct 27 04:18 (140.2h) [2026-10-21 08:08:54][INFO][training.pretrain_engine] Starting global step 34 (accumulating 32 micro-batches)... [2026-10-21 08:08:54][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 08:09:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9871 [2026-10-21 08:09:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0736 [2026-10-21 08:09:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2178 [2026-10-21 08:09:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7256 [2026-10-21 08:09:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7172 [2026-10-21 08:09:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5855 [2026-10-21 08:09:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4306 [2026-10-21 08:09:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6299 [2026-10-21 08:09:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6564 [2026-10-21 08:09:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3307 [2026-10-21 08:09:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6252 [2026-10-21 08:09:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5005 [2026-10-21 08:10:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1534 [2026-10-21 08:10:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8048 [2026-10-21 08:10:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3563 [2026-10-21 08:10:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2319 [2026-10-21 08:10:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6038 [2026-10-21 08:10:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4286 [2026-10-21 08:10:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8046 [2026-10-21 08:10:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5550 [2026-10-21 08:10:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8988 [2026-10-21 08:10:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4997 [2026-10-21 08:10:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5470 [2026-10-21 08:10:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.5123 [2026-10-21 08:10:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5633 [2026-10-21 08:10:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0552 [2026-10-21 08:10:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6581 [2026-10-21 08:10:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7787 [2026-10-21 08:11:12][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,622 | MFU: 1.8% [2026-10-21 08:11:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7634 [2026-10-21 08:11:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4120 [2026-10-21 08:11:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2063 [2026-10-21 08:11:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4274 [2026-10-21 08:11:17][INFO][training.pretrain_engine] [Ph2|Step 34] loss=5.3606 | ppl=212.85 | aux=0.02507 | lr=3.00e-04 | gnorm=0.484 | tok/s=29,187 | MFU=7.0% | step=143706ms | tokens=0.838B | mem=4.2/32.7GB | cost=$2.81/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 22:58 (134.8h) | Phase3: SKIPPED | SFT: Oct 27 03:43 (139.5h) | DPO (Final): Oct 27 04:18 (140.1h) [2026-10-21 08:11:17][INFO][training.pretrain_engine] Starting global step 35 (accumulating 32 micro-batches)... [2026-10-21 08:11:17][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 08:11:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4396 [2026-10-21 08:11:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4564 [2026-10-21 08:11:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1363 [2026-10-21 08:11:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2154 [2026-10-21 08:11:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8155 [2026-10-21 08:11:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0109 [2026-10-21 08:11:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8268 [2026-10-21 08:11:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3980 [2026-10-21 08:12:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8506 [2026-10-21 08:12:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3872 [2026-10-21 08:12:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6008 [2026-10-21 08:12:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5748 [2026-10-21 08:12:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3076 [2026-10-21 08:12:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8243 [2026-10-21 08:12:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5047 [2026-10-21 08:12:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0223 [2026-10-21 08:12:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6471 [2026-10-21 08:12:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3381 [2026-10-21 08:12:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3061 [2026-10-21 08:12:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6692 [2026-10-21 08:13:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2775 [2026-10-21 08:13:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6337 [2026-10-21 08:13:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8418 [2026-10-21 08:13:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2305 [2026-10-21 08:13:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8154 [2026-10-21 08:13:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9199 [2026-10-21 08:13:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0917 [2026-10-21 08:13:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6206 [2026-10-21 08:13:34][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,637 | MFU: 1.8% [2026-10-21 08:13:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4789 [2026-10-21 08:13:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3864 [2026-10-21 08:13:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1130 [2026-10-21 08:13:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9115 [2026-10-21 08:13:39][INFO][training.pretrain_engine] [Ph2|Step 35] loss=5.3239 | ppl=205.18 | aux=0.02507 | lr=3.00e-04 | gnorm=0.504 | tok/s=29,195 | MFU=7.0% | step=143666ms | tokens=0.839B | mem=4.2/32.7GB | cost=$2.89/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 22:57 (134.7h) | Phase3: SKIPPED | SFT: Oct 27 03:43 (139.5h) | DPO (Final): Oct 27 04:17 (140.1h) [2026-10-21 08:13:39][INFO][training.pretrain_engine] Starting global step 36 (accumulating 32 micro-batches)... [2026-10-21 08:13:39][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 08:13:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8517 [2026-10-21 08:13:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5391 [2026-10-21 08:13:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9224 [2026-10-21 08:13:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2962 [2026-10-21 08:14:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5721 [2026-10-21 08:14:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6545 [2026-10-21 08:14:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6745 [2026-10-21 08:14:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6204 [2026-10-21 08:14:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4877 [2026-10-21 08:14:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1451 [2026-10-21 08:14:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0818 [2026-10-21 08:14:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2721 [2026-10-21 08:14:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9330 [2026-10-21 08:14:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1099 [2026-10-21 08:14:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6942 [2026-10-21 08:14:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8637 [2026-10-21 08:15:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8458 [2026-10-21 08:15:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2946 [2026-10-21 08:15:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2298 [2026-10-21 08:15:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1575 [2026-10-21 08:15:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2290 [2026-10-21 08:15:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2297 [2026-10-21 08:15:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5238 [2026-10-21 08:15:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5591 [2026-10-21 08:15:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5509 [2026-10-21 08:15:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6628 [2026-10-21 08:15:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9603 [2026-10-21 08:15:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6021 [2026-10-21 08:15:57][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,632 | MFU: 1.8% [2026-10-21 08:16:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3111 [2026-10-21 08:16:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0976 [2026-10-21 08:16:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7732 [2026-10-21 08:16:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3005 [2026-10-21 08:16:02][INFO][training.pretrain_engine] [Ph2|Step 36] loss=5.3973 | ppl=220.82 | aux=0.02508 | lr=3.00e-04 | gnorm=0.551 | tok/s=29,202 | MFU=7.0% | step=143630ms | tokens=0.840B | mem=4.2/32.7GB | cost=$2.97/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 22:57 (134.7h) | Phase3: SKIPPED | SFT: Oct 27 03:43 (139.5h) | DPO (Final): Oct 27 04:17 (140.0h) [2026-10-21 08:16:05][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step36 | Tokens: 0.840B | Spent: $2.97 | Remaining: $997.03 [2026-10-21 08:16:05][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step36 [2026-10-21 08:16:05][INFO][training.pretrain_engine] Starting global step 37 (accumulating 32 micro-batches)... [2026-10-21 08:16:05][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 08:16:05][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 08:16:05][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 08:16:05][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 08:16:05][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step36 to Phase-Technologies/forge-500m-pretrain (path: phase2_step36)... [2026-10-21 08:16:05][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 08:16:16][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 08:16:16][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step36 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 08:16:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9055 [2026-10-21 08:16:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1536 [2026-10-21 08:16:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3422 [2026-10-21 08:16:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8169 [2026-10-21 08:16:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6143 [2026-10-21 08:16:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4036 [2026-10-21 08:16:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3892 [2026-10-21 08:16:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2604 [2026-10-21 08:16:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8745 [2026-10-21 08:16:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0251 [2026-10-21 08:16:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0869 [2026-10-21 08:16:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1075 [2026-10-21 08:17:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4644 [2026-10-21 08:17:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4944 [2026-10-21 08:17:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6049 [2026-10-21 08:17:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4649 [2026-10-21 08:17:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8336 [2026-10-21 08:17:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5447 [2026-10-21 08:17:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5696 [2026-10-21 08:17:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6676 [2026-10-21 08:17:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5591 [2026-10-21 08:17:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1875 [2026-10-21 08:17:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9102 [2026-10-21 08:17:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9518 [2026-10-21 08:18:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1733 [2026-10-21 08:18:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9255 [2026-10-21 08:18:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2690 [2026-10-21 08:18:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6702 [2026-10-21 08:18:23][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,625 | MFU: 1.8% [2026-10-21 08:18:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4151 [2026-10-21 08:18:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4738 [2026-10-21 08:18:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2658 [2026-10-21 08:18:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4419 [2026-10-21 08:18:27][INFO][training.pretrain_engine] [Ph2|Step 37] loss=5.4024 | ppl=221.93 | aux=0.02507 | lr=3.00e-04 | gnorm=0.605 | tok/s=29,211 | MFU=7.0% | step=143585ms | tokens=0.841B | mem=4.2/32.7GB | cost=$3.05/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 22:56 (134.6h) | Phase3: SKIPPED | SFT: Oct 27 03:42 (139.4h) | DPO (Final): Oct 27 04:16 (140.0h) [2026-10-21 08:18:27][INFO][training.pretrain_engine] Starting global step 38 (accumulating 32 micro-batches)... [2026-10-21 08:18:27][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 08:18:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2750 [2026-10-21 08:18:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5196 [2026-10-21 08:18:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4540 [2026-10-21 08:18:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5318 [2026-10-21 08:19:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7959 [2026-10-21 08:19:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9518 [2026-10-21 08:19:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0743 [2026-10-21 08:19:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8826 [2026-10-21 08:19:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7963 [2026-10-21 08:19:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5758 [2026-10-21 08:19:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0322 [2026-10-21 08:19:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2764 [2026-10-21 08:19:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4467 [2026-10-21 08:19:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2516 [2026-10-21 08:19:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5074 [2026-10-21 08:19:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4459 [2026-10-21 08:19:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9103 [2026-10-21 08:19:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7574 [2026-10-21 08:19:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1501 [2026-10-21 08:19:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7526 [2026-10-21 08:20:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7286 [2026-10-21 08:20:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9210 [2026-10-21 08:20:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8286 [2026-10-21 08:20:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6322 [2026-10-21 08:20:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5057 [2026-10-21 08:20:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7908 [2026-10-21 08:20:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2740 [2026-10-21 08:20:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3837 [2026-10-21 08:20:44][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,628 | MFU: 1.8% [2026-10-21 08:20:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5579 [2026-10-21 08:20:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1116 [2026-10-21 08:20:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0943 [2026-10-21 08:20:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.1101 [2026-10-21 08:20:50][INFO][training.pretrain_engine] [Ph2|Step 38] loss=5.3700 | ppl=214.86 | aux=0.02507 | lr=3.00e-04 | gnorm=0.750 | tok/s=29,217 | MFU=7.0% | step=143557ms | tokens=0.842B | mem=4.2/32.7GB | cost=$3.13/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 22:57 (134.6h) | Phase3: SKIPPED | SFT: Oct 27 03:42 (139.4h) | DPO (Final): Oct 27 04:16 (139.9h) [2026-10-21 08:20:50][INFO][training.pretrain_engine] Starting global step 39 (accumulating 32 micro-batches)... [2026-10-21 08:20:50][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 08:21:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1581 [2026-10-21 08:21:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5493 [2026-10-21 08:21:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3123 [2026-10-21 08:21:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6551 [2026-10-21 08:21:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4617 [2026-10-21 08:21:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1612 [2026-10-21 08:21:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5719 [2026-10-21 08:21:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2785 [2026-10-21 08:21:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8634 [2026-10-21 08:21:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3839 [2026-10-21 08:21:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9461 [2026-10-21 08:21:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7201 [2026-10-21 08:22:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5441 [2026-10-21 08:22:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5579 [2026-10-21 08:22:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4574 [2026-10-21 08:22:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7389 [2026-10-21 08:22:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1476 [2026-10-21 08:22:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9856 [2026-10-21 08:22:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3891 [2026-10-21 08:22:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0843 [2026-10-21 08:22:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0007 [2026-10-21 08:22:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3181 [2026-10-21 08:22:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9386 [2026-10-21 08:22:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0105 [2026-10-21 08:22:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5491 [2026-10-21 08:22:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4631 [2026-10-21 08:22:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6858 [2026-10-21 08:22:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4058 [2026-10-21 08:23:07][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,644 | MFU: 1.8% [2026-10-21 08:23:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3320 [2026-10-21 08:23:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0905 [2026-10-21 08:23:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2255 [2026-10-21 08:23:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1506 [2026-10-21 08:23:12][INFO][training.pretrain_engine] [Ph2|Step 39] loss=5.3471 | ppl=210.01 | aux=0.02509 | lr=3.00e-04 | gnorm=0.848 | tok/s=29,224 | MFU=7.0% | step=143523ms | tokens=0.843B | mem=4.2/32.7GB | cost=$3.22/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 22:57 (134.6h) | Phase3: SKIPPED | SFT: Oct 27 03:42 (139.3h) | DPO (Final): Oct 27 04:16 (139.9h) [2026-10-21 08:23:12][INFO][training.pretrain_engine] Starting global step 40 (accumulating 32 micro-batches)... [2026-10-21 08:23:12][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 08:23:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2525 [2026-10-21 08:23:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4307 [2026-10-21 08:23:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5061 [2026-10-21 08:23:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2522 [2026-10-21 08:23:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5753 [2026-10-21 08:23:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4939 [2026-10-21 08:23:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1720 [2026-10-21 08:23:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5329 [2026-10-21 08:24:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5207 [2026-10-21 08:24:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9364 [2026-10-21 08:24:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8472 [2026-10-21 08:24:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5768 [2026-10-21 08:24:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2817 [2026-10-21 08:24:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0891 [2026-10-21 08:24:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3891 [2026-10-21 08:24:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4000 [2026-10-21 08:24:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5736 [2026-10-21 08:24:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1450 [2026-10-21 08:24:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0987 [2026-10-21 08:24:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5330 [2026-10-21 08:24:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4664 [2026-10-21 08:24:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8124 [2026-10-21 08:24:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1293 [2026-10-21 08:24:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5531 [2026-10-21 08:25:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9881 [2026-10-21 08:25:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0809 [2026-10-21 08:25:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8581 [2026-10-21 08:25:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0058 [2026-10-21 08:25:29][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,650 | MFU: 1.8% [2026-10-21 08:25:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8546 [2026-10-21 08:25:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4721 [2026-10-21 08:25:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5484 [2026-10-21 08:25:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2668 [2026-10-21 08:25:34][INFO][training.pretrain_engine] [Ph2|Step 40] loss=5.3749 | ppl=215.93 | aux=0.02507 | lr=3.00e-04 | gnorm=0.855 | tok/s=29,232 | MFU=7.0% | step=143485ms | tokens=0.844B | mem=4.2/32.7GB | cost=$3.30/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 22:56 (134.5h) | Phase3: SKIPPED | SFT: Oct 27 03:41 (139.3h) | DPO (Final): Oct 27 04:15 (139.8h) [2026-10-21 08:25:34][INFO][training.pretrain_engine] Starting global step 41 (accumulating 32 micro-batches)... [2026-10-21 08:25:34][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 08:25:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6655 [2026-10-21 08:25:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9646 [2026-10-21 08:25:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4168 [2026-10-21 08:25:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5645 [2026-10-21 08:26:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8396 [2026-10-21 08:26:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3891 [2026-10-21 08:26:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6678 [2026-10-21 08:26:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2011 [2026-10-21 08:26:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7150 [2026-10-21 08:26:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2301 [2026-10-21 08:26:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1820 [2026-10-21 08:26:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2178 [2026-10-21 08:26:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8968 [2026-10-21 08:26:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5269 [2026-10-21 08:26:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7478 [2026-10-21 08:26:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0073 [2026-10-21 08:27:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6007 [2026-10-21 08:27:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8136 [2026-10-21 08:27:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2681 [2026-10-21 08:27:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8491 [2026-10-21 08:27:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4001 [2026-10-21 08:27:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1958 [2026-10-21 08:27:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3351 [2026-10-21 08:27:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2258 [2026-10-21 08:27:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7776 [2026-10-21 08:27:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5910 [2026-10-21 08:27:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1289 [2026-10-21 08:27:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6419 [2026-10-21 08:27:51][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,632 | MFU: 1.8% [2026-10-21 08:27:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2357 [2026-10-21 08:27:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5584 [2026-10-21 08:27:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0837 [2026-10-21 08:27:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7966 [2026-10-21 08:27:56][INFO][training.pretrain_engine] [Ph2|Step 41] loss=5.4654 | ppl=236.37 | aux=0.02507 | lr=3.00e-04 | gnorm=0.953 | tok/s=29,237 | MFU=7.0% | step=143456ms | tokens=0.845B | mem=4.2/32.7GB | cost=$3.38/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 22:56 (134.5h) | Phase3: SKIPPED | SFT: Oct 27 03:41 (139.2h) | DPO (Final): Oct 27 04:16 (139.8h) [2026-10-21 08:27:56][INFO][training.pretrain_engine] Starting global step 42 (accumulating 32 micro-batches)... [2026-10-21 08:27:56][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 08:28:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6616 [2026-10-21 08:28:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5140 [2026-10-21 08:28:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7262 [2026-10-21 08:28:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6388 [2026-10-21 08:28:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8826 [2026-10-21 08:28:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8834 [2026-10-21 08:28:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4437 [2026-10-21 08:28:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5878 [2026-10-21 08:28:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5087 [2026-10-21 08:28:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2788 [2026-10-21 08:28:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3230 [2026-10-21 08:28:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4395 [2026-10-21 08:29:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4084 [2026-10-21 08:29:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3066 [2026-10-21 08:29:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2547 [2026-10-21 08:29:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5187 [2026-10-21 08:29:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1956 [2026-10-21 08:29:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6203 [2026-10-21 08:29:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2438 [2026-10-21 08:29:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6371 [2026-10-21 08:29:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1854 [2026-10-21 08:29:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2351 [2026-10-21 08:29:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1661 [2026-10-21 08:29:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2010 [2026-10-21 08:30:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4357 [2026-10-21 08:30:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1425 [2026-10-21 08:30:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6202 [2026-10-21 08:30:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5017 [2026-10-21 08:30:13][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,654 | MFU: 1.8% [2026-10-21 08:30:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4967 [2026-10-21 08:30:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3347 [2026-10-21 08:30:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4974 [2026-10-21 08:30:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4855 [2026-10-21 08:30:18][INFO][training.pretrain_engine] [Ph2|Step 42] loss=5.4077 | ppl=223.11 | aux=0.02508 | lr=3.00e-04 | gnorm=1.469 | tok/s=29,243 | MFU=7.0% | step=143427ms | tokens=0.846B | mem=4.2/32.7GB | cost=$3.46/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 22:56 (134.4h) | Phase3: SKIPPED | SFT: Oct 27 03:41 (139.2h) | DPO (Final): Oct 27 04:16 (139.8h) [2026-10-21 08:30:18][INFO][training.pretrain_engine] Starting global step 43 (accumulating 32 micro-batches)... [2026-10-21 08:30:18][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 08:30:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0714 [2026-10-21 08:30:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5418 [2026-10-21 08:30:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5406 [2026-10-21 08:30:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7533 [2026-10-21 08:30:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5264 [2026-10-21 08:30:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5170 [2026-10-21 08:30:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5656 [2026-10-21 08:30:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3112 [2026-10-21 08:31:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0411 [2026-10-21 08:31:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1239 [2026-10-21 08:31:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1820 [2026-10-21 08:31:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6695 [2026-10-21 08:31:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2949 [2026-10-21 08:31:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0213 [2026-10-21 08:31:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9780 [2026-10-21 08:31:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7221 [2026-10-21 08:31:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2885 [2026-10-21 08:31:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4944 [2026-10-21 08:31:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3706 [2026-10-21 08:31:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1454 [2026-10-21 08:32:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5901 [2026-10-21 08:32:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3535 [2026-10-21 08:32:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8311 [2026-10-21 08:32:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4144 [2026-10-21 08:32:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2458 [2026-10-21 08:32:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2492 [2026-10-21 08:32:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4239 [2026-10-21 08:32:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3393 [2026-10-21 08:32:35][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,646 | MFU: 1.8% [2026-10-21 08:32:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9978 [2026-10-21 08:32:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7939 [2026-10-21 08:32:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0900 [2026-10-21 08:32:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5173 [2026-10-21 08:32:40][INFO][training.pretrain_engine] [Ph2|Step 43] loss=5.2567 | ppl=191.84 | aux=0.02508 | lr=3.00e-04 | gnorm=0.773 | tok/s=29,251 | MFU=7.0% | step=143391ms | tokens=0.847B | mem=4.2/32.7GB | cost=$3.54/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 22:56 (134.4h) | Phase3: SKIPPED | SFT: Oct 27 03:41 (139.1h) | DPO (Final): Oct 27 04:15 (139.7h) [2026-10-21 08:32:40][INFO][training.pretrain_engine] Starting global step 44 (accumulating 32 micro-batches)... [2026-10-21 08:32:40][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 08:32:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5706 [2026-10-21 08:32:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2932 [2026-10-21 08:32:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3791 [2026-10-21 08:32:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4364 [2026-10-21 08:33:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7335 [2026-10-21 08:33:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3886 [2026-10-21 08:33:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7810 [2026-10-21 08:33:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9361 [2026-10-21 08:33:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0733 [2026-10-21 08:33:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9928 [2026-10-21 08:33:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8007 [2026-10-21 08:33:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6578 [2026-10-21 08:33:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1123 [2026-10-21 08:33:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0931 [2026-10-21 08:33:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9984 [2026-10-21 08:33:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4556 [2026-10-21 08:34:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3714 [2026-10-21 08:34:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8393 [2026-10-21 08:34:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6314 [2026-10-21 08:34:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5949 [2026-10-21 08:34:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7187 [2026-10-21 08:34:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7558 [2026-10-21 08:34:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.3736 [2026-10-21 08:34:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6367 [2026-10-21 08:34:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3867 [2026-10-21 08:34:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1194 [2026-10-21 08:34:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5934 [2026-10-21 08:34:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1914 [2026-10-21 08:34:58][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,630 | MFU: 1.8% [2026-10-21 08:35:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1141 [2026-10-21 08:35:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7613 [2026-10-21 08:35:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5712 [2026-10-21 08:35:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7164 [2026-10-21 08:35:03][INFO][training.pretrain_engine] [Ph2|Step 44] loss=5.4200 | ppl=225.88 | aux=0.02507 | lr=3.00e-04 | gnorm=1.250 | tok/s=29,255 | MFU=7.0% | step=143369ms | tokens=0.848B | mem=4.2/32.7GB | cost=$3.62/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 22:57 (134.4h) | Phase3: SKIPPED | SFT: Oct 27 03:42 (139.1h) | DPO (Final): Oct 27 04:16 (139.7h) [2026-10-21 08:35:03][INFO][training.pretrain_engine] Starting global step 45 (accumulating 32 micro-batches)... [2026-10-21 08:35:03][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 08:35:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0625 [2026-10-21 08:35:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3305 [2026-10-21 08:35:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1030 [2026-10-21 08:35:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2471 [2026-10-21 08:35:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7718 [2026-10-21 08:35:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1850 [2026-10-21 08:35:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2440 [2026-10-21 08:35:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2831 [2026-10-21 08:35:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3289 [2026-10-21 08:35:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4541 [2026-10-21 08:35:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2120 [2026-10-21 08:35:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4743 [2026-10-21 08:36:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2978 [2026-10-21 08:36:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2332 [2026-10-21 08:36:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2741 [2026-10-21 08:36:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1725 [2026-10-21 08:36:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3638 [2026-10-21 08:36:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6421 [2026-10-21 08:36:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1598 [2026-10-21 08:36:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3635 [2026-10-21 08:36:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=3.7365 [2026-10-21 08:36:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8039 [2026-10-21 08:36:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1548 [2026-10-21 08:36:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4768 [2026-10-21 08:37:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8082 [2026-10-21 08:37:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3463 [2026-10-21 08:37:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1721 [2026-10-21 08:37:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9784 [2026-10-21 08:37:20][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,620 | MFU: 1.8% [2026-10-21 08:37:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3070 [2026-10-21 08:37:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5536 [2026-10-21 08:37:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4846 [2026-10-21 08:37:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7115 [2026-10-21 08:37:25][INFO][training.pretrain_engine] [Ph2|Step 45] loss=5.3300 | ppl=206.43 | aux=0.02508 | lr=3.00e-04 | gnorm=0.883 | tok/s=29,259 | MFU=7.0% | step=143350ms | tokens=0.849B | mem=4.2/32.7GB | cost=$3.70/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 22:57 (134.3h) | Phase3: SKIPPED | SFT: Oct 27 03:42 (139.1h) | DPO (Final): Oct 27 04:16 (139.7h) [2026-10-21 08:37:28][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step45 | Tokens: 0.849B | Spent: $3.71 | Remaining: $996.29 [2026-10-21 08:37:28][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step45 [2026-10-21 08:37:28][INFO][training.pretrain_engine] Starting global step 46 (accumulating 32 micro-batches)... [2026-10-21 08:37:28][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 08:37:28][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 08:37:28][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 08:37:29][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 08:37:29][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step45 to Phase-Technologies/forge-500m-pretrain (path: phase2_step45)... [2026-10-21 08:37:29][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 08:37:40][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 08:37:40][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step45 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 08:37:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4169 [2026-10-21 08:37:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8378 [2026-10-21 08:37:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7998 [2026-10-21 08:37:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4030 [2026-10-21 08:38:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3868 [2026-10-21 08:38:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3637 [2026-10-21 08:38:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5293 [2026-10-21 08:38:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9165 [2026-10-21 08:38:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5162 [2026-10-21 08:38:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.0305 [2026-10-21 08:38:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7163 [2026-10-21 08:38:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9118 [2026-10-21 08:38:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9934 [2026-10-21 08:38:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5872 [2026-10-21 08:38:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2433 [2026-10-21 08:38:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6136 [2026-10-21 08:38:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4254 [2026-10-21 08:38:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5253 [2026-10-21 08:38:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0055 [2026-10-21 08:38:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4184 [2026-10-21 08:39:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5226 [2026-10-21 08:39:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3320 [2026-10-21 08:39:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9804 [2026-10-21 08:39:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5756 [2026-10-21 08:39:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3603 [2026-10-21 08:39:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5953 [2026-10-21 08:39:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9919 [2026-10-21 08:39:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2164 [2026-10-21 08:39:46][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,632 | MFU: 1.8% [2026-10-21 08:39:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3904 [2026-10-21 08:39:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4439 [2026-10-21 08:39:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5989 [2026-10-21 08:39:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2988 [2026-10-21 08:39:50][INFO][training.pretrain_engine] [Ph2|Step 46] loss=5.4063 | ppl=222.81 | aux=0.02508 | lr=3.00e-04 | gnorm=1.109 | tok/s=29,266 | MFU=7.0% | step=143318ms | tokens=0.850B | mem=4.2/32.7GB | cost=$3.79/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 22:57 (134.3h) | Phase3: SKIPPED | SFT: Oct 27 03:42 (139.0h) | DPO (Final): Oct 27 04:16 (139.6h) [2026-10-21 08:39:50][INFO][training.pretrain_engine] Starting global step 47 (accumulating 32 micro-batches)... [2026-10-21 08:39:50][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 08:40:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8621 [2026-10-21 08:40:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0115 [2026-10-21 08:40:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9938 [2026-10-21 08:40:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6728 [2026-10-21 08:40:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6497 [2026-10-21 08:40:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0819 [2026-10-21 08:40:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3196 [2026-10-21 08:40:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6582 [2026-10-21 08:40:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4241 [2026-10-21 08:40:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8755 [2026-10-21 08:40:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1376 [2026-10-21 08:40:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1409 [2026-10-21 08:41:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5228 [2026-10-21 08:41:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4416 [2026-10-21 08:41:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4621 [2026-10-21 08:41:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8736 [2026-10-21 08:41:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.3024 [2026-10-21 08:41:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9034 [2026-10-21 08:41:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1740 [2026-10-21 08:41:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3237 [2026-10-21 08:41:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4252 [2026-10-21 08:41:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3755 [2026-10-21 08:41:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0958 [2026-10-21 08:41:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3960 [2026-10-21 08:41:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3409 [2026-10-21 08:41:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7301 [2026-10-21 08:41:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3095 [2026-10-21 08:41:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9505 [2026-10-21 08:42:08][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,619 | MFU: 1.8% [2026-10-21 08:42:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1501 [2026-10-21 08:42:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8417 [2026-10-21 08:42:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4032 [2026-10-21 08:42:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2504 [2026-10-21 08:42:13][INFO][training.pretrain_engine] [Ph2|Step 47] loss=5.3185 | ppl=204.07 | aux=0.02507 | lr=3.00e-04 | gnorm=0.789 | tok/s=29,269 | MFU=7.0% | step=143304ms | tokens=0.851B | mem=4.2/32.7GB | cost=$3.87/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 22:58 (134.3h) | Phase3: SKIPPED | SFT: Oct 27 03:43 (139.0h) | DPO (Final): Oct 27 04:17 (139.6h) [2026-10-21 08:42:13][INFO][training.pretrain_engine] Starting global step 48 (accumulating 32 micro-batches)... [2026-10-21 08:42:13][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 08:42:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5273 [2026-10-21 08:42:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8284 [2026-10-21 08:42:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5540 [2026-10-21 08:42:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5795 [2026-10-21 08:42:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2908 [2026-10-21 08:42:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2817 [2026-10-21 08:42:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5945 [2026-10-21 08:42:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5511 [2026-10-21 08:43:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2488 [2026-10-21 08:43:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4291 [2026-10-21 08:43:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2842 [2026-10-21 08:43:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2500 [2026-10-21 08:43:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7185 [2026-10-21 08:43:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4144 [2026-10-21 08:43:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6297 [2026-10-21 08:43:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7664 [2026-10-21 08:43:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4821 [2026-10-21 08:43:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4238 [2026-10-21 08:43:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3679 [2026-10-21 08:43:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0404 [2026-10-21 08:43:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5255 [2026-10-21 08:43:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4437 [2026-10-21 08:44:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7964 [2026-10-21 08:44:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6328 [2026-10-21 08:44:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4668 [2026-10-21 08:44:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0178 [2026-10-21 08:44:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3371 [2026-10-21 08:44:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8481 [2026-10-21 08:44:31][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,610 | MFU: 1.8% [2026-10-21 08:44:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5126 [2026-10-21 08:44:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3153 [2026-10-21 08:44:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2614 [2026-10-21 08:44:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3864 [2026-10-21 08:44:36][INFO][training.pretrain_engine] [Ph2|Step 48] loss=5.2853 | ppl=197.42 | aux=0.02507 | lr=3.00e-04 | gnorm=0.844 | tok/s=29,271 | MFU=7.0% | step=143294ms | tokens=0.852B | mem=4.2/32.7GB | cost=$3.95/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:00 (134.3h) | Phase3: SKIPPED | SFT: Oct 27 03:44 (139.0h) | DPO (Final): Oct 27 04:19 (139.6h) [2026-10-21 08:44:36][INFO][training.pretrain_engine] Starting global step 49 (accumulating 32 micro-batches)... [2026-10-21 08:44:36][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 08:44:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6032 [2026-10-21 08:44:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5753 [2026-10-21 08:44:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8203 [2026-10-21 08:44:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6074 [2026-10-21 08:45:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4553 [2026-10-21 08:45:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6335 [2026-10-21 08:45:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1123 [2026-10-21 08:45:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3946 [2026-10-21 08:45:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6357 [2026-10-21 08:45:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5284 [2026-10-21 08:45:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5785 [2026-10-21 08:45:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9073 [2026-10-21 08:45:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6307 [2026-10-21 08:45:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6215 [2026-10-21 08:45:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6582 [2026-10-21 08:45:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4962 [2026-10-21 08:46:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2299 [2026-10-21 08:46:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2443 [2026-10-21 08:46:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1558 [2026-10-21 08:46:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1297 [2026-10-21 08:46:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7706 [2026-10-21 08:46:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7238 [2026-10-21 08:46:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9578 [2026-10-21 08:46:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8575 [2026-10-21 08:46:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4387 [2026-10-21 08:46:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7510 [2026-10-21 08:46:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1388 [2026-10-21 08:46:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0167 [2026-10-21 08:46:53][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,612 | MFU: 1.8% [2026-10-21 08:46:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2320 [2026-10-21 08:46:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2520 [2026-10-21 08:46:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8780 [2026-10-21 08:46:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5668 [2026-10-21 08:46:59][INFO][training.pretrain_engine] [Ph2|Step 49] loss=5.4231 | ppl=226.57 | aux=0.02507 | lr=3.00e-04 | gnorm=0.750 | tok/s=29,272 | MFU=7.0% | step=143289ms | tokens=0.854B | mem=4.2/32.7GB | cost=$4.03/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:01 (134.2h) | Phase3: SKIPPED | SFT: Oct 27 03:46 (139.0h) | DPO (Final): Oct 27 04:20 (139.6h) [2026-10-21 08:46:59][INFO][training.pretrain_engine] Starting global step 50 (accumulating 32 micro-batches)... [2026-10-21 08:46:59][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 08:47:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2839 [2026-10-21 08:47:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5473 [2026-10-21 08:47:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5806 [2026-10-21 08:47:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3612 [2026-10-21 08:47:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0927 [2026-10-21 08:47:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1366 [2026-10-21 08:47:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2894 [2026-10-21 08:47:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0121 [2026-10-21 08:47:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1264 [2026-10-21 08:47:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5813 [2026-10-21 08:47:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9003 [2026-10-21 08:47:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9105 [2026-10-21 08:48:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5836 [2026-10-21 08:48:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1592 [2026-10-21 08:48:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9536 [2026-10-21 08:48:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5073 [2026-10-21 08:48:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6035 [2026-10-21 08:48:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4680 [2026-10-21 08:48:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3253 [2026-10-21 08:48:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5350 [2026-10-21 08:48:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1020 [2026-10-21 08:48:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2412 [2026-10-21 08:48:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5129 [2026-10-21 08:48:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4349 [2026-10-21 08:49:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4641 [2026-10-21 08:49:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2482 [2026-10-21 08:49:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9701 [2026-10-21 08:49:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8505 [2026-10-21 08:49:16][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,630 | MFU: 1.8% [2026-10-21 08:49:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5815 [2026-10-21 08:49:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4334 [2026-10-21 08:49:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7023 [2026-10-21 08:49:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6545 [2026-10-21 08:49:21][INFO][training.pretrain_engine] [Ph2|Step 50] loss=5.3549 | ppl=211.64 | aux=0.02506 | lr=3.00e-04 | gnorm=0.840 | tok/s=29,276 | MFU=7.0% | step=143269ms | tokens=0.855B | mem=4.2/32.7GB | cost=$4.11/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:02 (134.2h) | Phase3: SKIPPED | SFT: Oct 27 03:47 (139.0h) | DPO (Final): Oct 27 04:21 (139.5h) [2026-10-21 08:49:21][INFO][training.pretrain_engine] Starting global step 51 (accumulating 32 micro-batches)... [2026-10-21 08:49:21][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 08:49:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6327 [2026-10-21 08:49:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4219 [2026-10-21 08:49:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4773 [2026-10-21 08:49:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5578 [2026-10-21 08:49:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4697 [2026-10-21 08:49:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6826 [2026-10-21 08:49:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4016 [2026-10-21 08:49:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5435 [2026-10-21 08:50:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4869 [2026-10-21 08:50:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2805 [2026-10-21 08:50:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3528 [2026-10-21 08:50:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1149 [2026-10-21 08:50:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4409 [2026-10-21 08:50:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3366 [2026-10-21 08:50:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3546 [2026-10-21 08:50:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7021 [2026-10-21 08:50:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0253 [2026-10-21 08:50:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6114 [2026-10-21 08:50:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2494 [2026-10-21 08:50:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1555 [2026-10-21 08:51:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4232 [2026-10-21 08:51:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4291 [2026-10-21 08:51:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3325 [2026-10-21 08:51:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3290 [2026-10-21 08:51:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9991 [2026-10-21 08:51:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5401 [2026-10-21 08:51:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2691 [2026-10-21 08:51:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5445 [2026-10-21 08:51:38][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,633 | MFU: 1.8% [2026-10-21 08:51:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5243 [2026-10-21 08:51:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5339 [2026-10-21 08:51:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4823 [2026-10-21 08:51:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2054 [2026-10-21 08:51:43][INFO][training.pretrain_engine] [Ph2|Step 51] loss=5.3673 | ppl=214.28 | aux=0.02507 | lr=3.00e-04 | gnorm=0.824 | tok/s=29,279 | MFU=7.0% | step=143252ms | tokens=0.856B | mem=4.2/32.7GB | cost=$4.20/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:03 (134.2h) | Phase3: SKIPPED | SFT: Oct 27 03:47 (138.9h) | DPO (Final): Oct 27 04:21 (139.5h) [2026-10-21 08:51:43][INFO][training.pretrain_engine] Starting global step 52 (accumulating 32 micro-batches)... [2026-10-21 08:51:43][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 08:52:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2262 [2026-10-21 08:52:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0868 [2026-10-21 08:52:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8929 [2026-10-21 08:52:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2533 [2026-10-21 08:52:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8530 [2026-10-21 08:52:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9885 [2026-10-21 08:52:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3713 [2026-10-21 08:52:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1014 [2026-10-21 08:52:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5963 [2026-10-21 08:52:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1805 [2026-10-21 08:52:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2216 [2026-10-21 08:52:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2383 [2026-10-21 08:52:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3362 [2026-10-21 08:52:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.1804 [2026-10-21 08:52:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2203 [2026-10-21 08:52:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3290 [2026-10-21 08:53:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5780 [2026-10-21 08:53:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0953 [2026-10-21 08:53:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8156 [2026-10-21 08:53:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4084 [2026-10-21 08:53:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9802 [2026-10-21 08:53:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3886 [2026-10-21 08:53:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6034 [2026-10-21 08:53:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1790 [2026-10-21 08:53:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8507 [2026-10-21 08:53:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4896 [2026-10-21 08:53:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0926 [2026-10-21 08:53:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4398 [2026-10-21 08:54:01][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,622 | MFU: 1.8% [2026-10-21 08:54:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2772 [2026-10-21 08:54:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4830 [2026-10-21 08:54:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3487 [2026-10-21 08:54:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4549 [2026-10-21 08:54:06][INFO][training.pretrain_engine] [Ph2|Step 52] loss=5.2190 | ppl=184.74 | aux=0.02507 | lr=3.00e-04 | gnorm=0.680 | tok/s=29,281 | MFU=7.0% | step=143245ms | tokens=0.857B | mem=4.2/32.7GB | cost=$4.28/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:04 (134.2h) | Phase3: SKIPPED | SFT: Oct 27 03:49 (138.9h) | DPO (Final): Oct 27 04:23 (139.5h) [2026-10-21 08:54:06][INFO][training.pretrain_engine] Starting global step 53 (accumulating 32 micro-batches)... [2026-10-21 08:54:06][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 08:54:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4614 [2026-10-21 08:54:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0868 [2026-10-21 08:54:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7710 [2026-10-21 08:54:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5619 [2026-10-21 08:54:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6755 [2026-10-21 08:54:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7221 [2026-10-21 08:54:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0991 [2026-10-21 08:54:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2724 [2026-10-21 08:54:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1040 [2026-10-21 08:54:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5916 [2026-10-21 08:55:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4858 [2026-10-21 08:55:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2926 [2026-10-21 08:55:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2780 [2026-10-21 08:55:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4301 [2026-10-21 08:55:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8358 [2026-10-21 08:55:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1805 [2026-10-21 08:55:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8034 [2026-10-21 08:55:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9782 [2026-10-21 08:55:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9785 [2026-10-21 08:55:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0907 [2026-10-21 08:55:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5659 [2026-10-21 08:55:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7230 [2026-10-21 08:55:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0604 [2026-10-21 08:55:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7553 [2026-10-21 08:56:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2166 [2026-10-21 08:56:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0062 [2026-10-21 08:56:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3189 [2026-10-21 08:56:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3816 [2026-10-21 08:56:24][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,630 | MFU: 1.8% [2026-10-21 08:56:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5161 [2026-10-21 08:56:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0384 [2026-10-21 08:56:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3347 [2026-10-21 08:56:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3830 [2026-10-21 08:56:29][INFO][training.pretrain_engine] [Ph2|Step 53] loss=5.2803 | ppl=196.42 | aux=0.02507 | lr=3.00e-04 | gnorm=0.656 | tok/s=29,283 | MFU=7.0% | step=143233ms | tokens=0.858B | mem=4.2/32.7GB | cost=$4.36/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:05 (134.2h) | Phase3: SKIPPED | SFT: Oct 27 03:50 (138.9h) | DPO (Final): Oct 27 04:24 (139.5h) [2026-10-21 08:56:29][INFO][training.pretrain_engine] Starting global step 54 (accumulating 32 micro-batches)... [2026-10-21 08:56:29][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 08:56:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0524 [2026-10-21 08:56:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1774 [2026-10-21 08:56:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1717 [2026-10-21 08:56:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2326 [2026-10-21 08:57:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3044 [2026-10-21 08:57:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5209 [2026-10-21 08:57:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8650 [2026-10-21 08:57:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6322 [2026-10-21 08:57:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7805 [2026-10-21 08:57:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4661 [2026-10-21 08:57:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3287 [2026-10-21 08:57:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1894 [2026-10-21 08:57:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1215 [2026-10-21 08:57:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0488 [2026-10-21 08:57:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8342 [2026-10-21 08:57:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7754 [2026-10-21 08:57:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4474 [2026-10-21 08:57:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6455 [2026-10-21 08:57:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4980 [2026-10-21 08:57:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5058 [2026-10-21 08:58:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1590 [2026-10-21 08:58:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9946 [2026-10-21 08:58:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0598 [2026-10-21 08:58:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6278 [2026-10-21 08:58:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4537 [2026-10-21 08:58:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0580 [2026-10-21 08:58:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7788 [2026-10-21 08:58:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6530 [2026-10-21 08:58:46][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,632 | MFU: 1.8% [2026-10-21 08:58:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9640 [2026-10-21 08:58:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6236 [2026-10-21 08:58:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8129 [2026-10-21 08:58:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2514 [2026-10-21 08:58:51][INFO][training.pretrain_engine] [Ph2|Step 54] loss=5.2533 | ppl=191.19 | aux=0.02507 | lr=3.00e-04 | gnorm=0.738 | tok/s=29,286 | MFU=7.0% | step=143219ms | tokens=0.859B | mem=4.2/32.7GB | cost=$4.44/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:06 (134.1h) | Phase3: SKIPPED | SFT: Oct 27 03:51 (138.9h) | DPO (Final): Oct 27 04:25 (139.4h) [2026-10-21 08:58:54][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step54 | Tokens: 0.859B | Spent: $4.44 | Remaining: $995.56 [2026-10-21 08:58:54][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step54 [2026-10-21 08:58:54][INFO][training.pretrain_engine] Starting global step 55 (accumulating 32 micro-batches)... [2026-10-21 08:58:54][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 08:58:55][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 08:58:55][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 08:58:55][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 08:58:55][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step54 to Phase-Technologies/forge-500m-pretrain (path: phase2_step54)... [2026-10-21 08:58:55][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 08:59:05][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 08:59:05][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step54 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 08:59:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4099 [2026-10-21 08:59:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4894 [2026-10-21 08:59:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3393 [2026-10-21 08:59:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4164 [2026-10-21 08:59:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8924 [2026-10-21 08:59:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6736 [2026-10-21 08:59:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1948 [2026-10-21 08:59:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0187 [2026-10-21 08:59:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0561 [2026-10-21 08:59:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6078 [2026-10-21 08:59:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8768 [2026-10-21 08:59:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5604 [2026-10-21 09:00:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1963 [2026-10-21 09:00:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2211 [2026-10-21 09:00:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2650 [2026-10-21 09:00:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2094 [2026-10-21 09:00:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8103 [2026-10-21 09:00:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0534 [2026-10-21 09:00:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9780 [2026-10-21 09:00:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5858 [2026-10-21 09:00:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5818 [2026-10-21 09:00:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6105 [2026-10-21 09:00:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6908 [2026-10-21 09:00:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7215 [2026-10-21 09:00:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6963 [2026-10-21 09:00:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4891 [2026-10-21 09:00:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6988 [2026-10-21 09:00:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5757 [2026-10-21 09:01:12][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,631 | MFU: 1.8% [2026-10-21 09:01:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4817 [2026-10-21 09:01:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4563 [2026-10-21 09:01:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2518 [2026-10-21 09:01:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1564 [2026-10-21 09:01:16][INFO][training.pretrain_engine] [Ph2|Step 55] loss=5.4250 | ppl=227.02 | aux=0.02507 | lr=3.00e-04 | gnorm=0.637 | tok/s=29,291 | MFU=7.0% | step=143195ms | tokens=0.860B | mem=4.2/32.7GB | cost=$4.52/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:07 (134.1h) | Phase3: SKIPPED | SFT: Oct 27 03:51 (138.8h) | DPO (Final): Oct 27 04:25 (139.4h) [2026-10-21 09:01:16][INFO][training.pretrain_engine] Starting global step 56 (accumulating 32 micro-batches)... [2026-10-21 09:01:16][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 09:01:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4930 [2026-10-21 09:01:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4798 [2026-10-21 09:01:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3166 [2026-10-21 09:01:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1632 [2026-10-21 09:01:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4429 [2026-10-21 09:01:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6150 [2026-10-21 09:01:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5408 [2026-10-21 09:01:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4035 [2026-10-21 09:02:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2310 [2026-10-21 09:02:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2729 [2026-10-21 09:02:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2485 [2026-10-21 09:02:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3672 [2026-10-21 09:02:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1364 [2026-10-21 09:02:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4765 [2026-10-21 09:02:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0824 [2026-10-21 09:02:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4257 [2026-10-21 09:02:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6895 [2026-10-21 09:02:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3069 [2026-10-21 09:02:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2509 [2026-10-21 09:02:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6886 [2026-10-21 09:03:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5620 [2026-10-21 09:03:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4000 [2026-10-21 09:03:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0823 [2026-10-21 09:03:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5611 [2026-10-21 09:03:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7817 [2026-10-21 09:03:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5099 [2026-10-21 09:03:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4185 [2026-10-21 09:03:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6758 [2026-10-21 09:03:34][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,635 | MFU: 1.8% [2026-10-21 09:03:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7591 [2026-10-21 09:03:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8022 [2026-10-21 09:03:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6197 [2026-10-21 09:03:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3747 [2026-10-21 09:03:39][INFO][training.pretrain_engine] [Ph2|Step 56] loss=5.4156 | ppl=224.89 | aux=0.02506 | lr=3.00e-04 | gnorm=0.590 | tok/s=29,293 | MFU=7.0% | step=143184ms | tokens=0.861B | mem=4.2/32.7GB | cost=$4.61/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:08 (134.1h) | Phase3: SKIPPED | SFT: Oct 27 03:52 (138.8h) | DPO (Final): Oct 27 04:26 (139.4h) [2026-10-21 09:03:39][INFO][training.pretrain_engine] Starting global step 57 (accumulating 32 micro-batches)... [2026-10-21 09:03:39][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 09:03:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3144 [2026-10-21 09:03:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2778 [2026-10-21 09:03:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4281 [2026-10-21 09:03:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4685 [2026-10-21 09:04:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4433 [2026-10-21 09:04:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2622 [2026-10-21 09:04:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6024 [2026-10-21 09:04:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0864 [2026-10-21 09:04:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8444 [2026-10-21 09:04:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8534 [2026-10-21 09:04:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4106 [2026-10-21 09:04:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8343 [2026-10-21 09:04:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1219 [2026-10-21 09:04:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5416 [2026-10-21 09:04:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3671 [2026-10-21 09:04:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6039 [2026-10-21 09:05:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1978 [2026-10-21 09:05:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3809 [2026-10-21 09:05:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6920 [2026-10-21 09:05:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5417 [2026-10-21 09:05:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3661 [2026-10-21 09:05:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4397 [2026-10-21 09:05:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0867 [2026-10-21 09:05:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4601 [2026-10-21 09:05:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4318 [2026-10-21 09:05:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2797 [2026-10-21 09:05:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9577 [2026-10-21 09:05:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3993 [2026-10-21 09:05:57][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,614 | MFU: 1.8% [2026-10-21 09:06:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5176 [2026-10-21 09:06:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1544 [2026-10-21 09:06:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6686 [2026-10-21 09:06:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3340 [2026-10-21 09:06:02][INFO][training.pretrain_engine] [Ph2|Step 57] loss=5.3727 | ppl=215.45 | aux=0.02506 | lr=3.00e-04 | gnorm=0.641 | tok/s=29,294 | MFU=7.0% | step=143179ms | tokens=0.862B | mem=4.2/32.7GB | cost=$4.69/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:09 (134.1h) | Phase3: SKIPPED | SFT: Oct 27 03:54 (138.8h) | DPO (Final): Oct 27 04:28 (139.4h) [2026-10-21 09:06:02][INFO][training.pretrain_engine] Starting global step 58 (accumulating 32 micro-batches)... [2026-10-21 09:06:02][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 09:06:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1145 [2026-10-21 09:06:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5921 [2026-10-21 09:06:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4661 [2026-10-21 09:06:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8573 [2026-10-21 09:06:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7997 [2026-10-21 09:06:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6650 [2026-10-21 09:06:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5491 [2026-10-21 09:06:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5163 [2026-10-21 09:06:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6432 [2026-10-21 09:06:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2594 [2026-10-21 09:06:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1398 [2026-10-21 09:06:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6323 [2026-10-21 09:07:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0371 [2026-10-21 09:07:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7753 [2026-10-21 09:07:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2553 [2026-10-21 09:07:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6530 [2026-10-21 09:07:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8635 [2026-10-21 09:07:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2682 [2026-10-21 09:07:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6054 [2026-10-21 09:07:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0732 [2026-10-21 09:07:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0162 [2026-10-21 09:07:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8389 [2026-10-21 09:07:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7598 [2026-10-21 09:07:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3843 [2026-10-21 09:08:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5678 [2026-10-21 09:08:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6344 [2026-10-21 09:08:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5932 [2026-10-21 09:08:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1840 [2026-10-21 09:08:20][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,608 | MFU: 1.8% [2026-10-21 09:08:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5393 [2026-10-21 09:08:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1956 [2026-10-21 09:08:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9476 [2026-10-21 09:08:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7107 [2026-10-21 09:08:25][INFO][training.pretrain_engine] [Ph2|Step 58] loss=5.3753 | ppl=216.01 | aux=0.02507 | lr=3.00e-04 | gnorm=0.570 | tok/s=29,294 | MFU=7.0% | step=143177ms | tokens=0.863B | mem=4.2/32.7GB | cost=$4.77/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:11 (134.1h) | Phase3: SKIPPED | SFT: Oct 27 03:55 (138.8h) | DPO (Final): Oct 27 04:30 (139.4h) [2026-10-21 09:08:25][INFO][training.pretrain_engine] Starting global step 59 (accumulating 32 micro-batches)... [2026-10-21 09:08:25][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 09:08:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5281 [2026-10-21 09:08:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1968 [2026-10-21 09:08:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3818 [2026-10-21 09:08:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9150 [2026-10-21 09:09:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1955 [2026-10-21 09:09:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5211 [2026-10-21 09:09:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5824 [2026-10-21 09:09:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4794 [2026-10-21 09:09:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2076 [2026-10-21 09:09:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7234 [2026-10-21 09:09:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3107 [2026-10-21 09:09:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0602 [2026-10-21 09:09:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4091 [2026-10-21 09:09:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5984 [2026-10-21 09:09:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9936 [2026-10-21 09:09:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0768 [2026-10-21 09:09:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2824 [2026-10-21 09:09:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4571 [2026-10-21 09:09:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3630 [2026-10-21 09:09:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1027 [2026-10-21 09:10:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6693 [2026-10-21 09:10:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6247 [2026-10-21 09:10:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8361 [2026-10-21 09:10:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1822 [2026-10-21 09:10:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3244 [2026-10-21 09:10:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3801 [2026-10-21 09:10:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4067 [2026-10-21 09:10:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4195 [2026-10-21 09:10:43][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,613 | MFU: 1.8% [2026-10-21 09:10:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.1780 [2026-10-21 09:10:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9726 [2026-10-21 09:10:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8061 [2026-10-21 09:10:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=3.9443 [2026-10-21 09:10:48][INFO][training.pretrain_engine] [Ph2|Step 59] loss=5.3080 | ppl=201.94 | aux=0.02506 | lr=3.00e-04 | gnorm=0.500 | tok/s=29,296 | MFU=7.0% | step=143170ms | tokens=0.864B | mem=4.2/32.7GB | cost=$4.85/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:12 (134.0h) | Phase3: SKIPPED | SFT: Oct 27 03:57 (138.8h) | DPO (Final): Oct 27 04:31 (139.3h) [2026-10-21 09:10:48][INFO][training.pretrain_engine] Starting global step 60 (accumulating 32 micro-batches)... [2026-10-21 09:10:48][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 09:11:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1800 [2026-10-21 09:11:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4703 [2026-10-21 09:11:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2660 [2026-10-21 09:11:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5372 [2026-10-21 09:11:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4937 [2026-10-21 09:11:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9800 [2026-10-21 09:11:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8767 [2026-10-21 09:11:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4288 [2026-10-21 09:11:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0930 [2026-10-21 09:11:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0058 [2026-10-21 09:11:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7935 [2026-10-21 09:11:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0079 [2026-10-21 09:11:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7942 [2026-10-21 09:11:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5867 [2026-10-21 09:11:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6716 [2026-10-21 09:11:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5787 [2026-10-21 09:12:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5604 [2026-10-21 09:12:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3335 [2026-10-21 09:12:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9283 [2026-10-21 09:12:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3823 [2026-10-21 09:12:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6205 [2026-10-21 09:12:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4302 [2026-10-21 09:12:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4213 [2026-10-21 09:12:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5635 [2026-10-21 09:12:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0926 [2026-10-21 09:12:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1736 [2026-10-21 09:12:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6360 [2026-10-21 09:12:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1864 [2026-10-21 09:13:05][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,619 | MFU: 1.8% [2026-10-21 09:13:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5167 [2026-10-21 09:13:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3601 [2026-10-21 09:13:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5796 [2026-10-21 09:13:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1892 [2026-10-21 09:13:11][INFO][training.pretrain_engine] [Ph2|Step 60] loss=5.3068 | ppl=201.71 | aux=0.02506 | lr=3.00e-04 | gnorm=0.516 | tok/s=29,297 | MFU=7.0% | step=143164ms | tokens=0.865B | mem=4.2/32.7GB | cost=$4.93/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:14 (134.0h) | Phase3: SKIPPED | SFT: Oct 27 03:58 (138.8h) | DPO (Final): Oct 27 04:32 (139.3h) [2026-10-21 09:13:11][INFO][training.pretrain_engine] Starting global step 61 (accumulating 32 micro-batches)... [2026-10-21 09:13:11][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 09:13:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4804 [2026-10-21 09:13:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6326 [2026-10-21 09:13:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7723 [2026-10-21 09:13:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5207 [2026-10-21 09:13:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5785 [2026-10-21 09:13:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4858 [2026-10-21 09:13:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4739 [2026-10-21 09:13:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5980 [2026-10-21 09:14:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3406 [2026-10-21 09:14:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5071 [2026-10-21 09:14:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5850 [2026-10-21 09:14:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8873 [2026-10-21 09:14:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4329 [2026-10-21 09:14:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2456 [2026-10-21 09:14:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4720 [2026-10-21 09:14:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8046 [2026-10-21 09:14:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0471 [2026-10-21 09:14:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0932 [2026-10-21 09:14:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9590 [2026-10-21 09:14:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9560 [2026-10-21 09:14:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5445 [2026-10-21 09:14:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6496 [2026-10-21 09:14:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9149 [2026-10-21 09:14:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4726 [2026-10-21 09:15:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6333 [2026-10-21 09:15:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6927 [2026-10-21 09:15:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4039 [2026-10-21 09:15:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3329 [2026-10-21 09:15:28][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,625 | MFU: 1.8% [2026-10-21 09:15:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3432 [2026-10-21 09:15:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3714 [2026-10-21 09:15:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1827 [2026-10-21 09:15:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2529 [2026-10-21 09:15:33][INFO][training.pretrain_engine] [Ph2|Step 61] loss=5.2851 | ppl=197.38 | aux=0.02507 | lr=3.00e-04 | gnorm=0.539 | tok/s=29,298 | MFU=7.0% | step=143158ms | tokens=0.866B | mem=4.2/32.7GB | cost=$5.01/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:15 (134.0h) | Phase3: SKIPPED | SFT: Oct 27 04:00 (138.7h) | DPO (Final): Oct 27 04:34 (139.3h) [2026-10-21 09:15:33][INFO][training.pretrain_engine] Starting global step 62 (accumulating 32 micro-batches)... [2026-10-21 09:15:33][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 09:15:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0566 [2026-10-21 09:15:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9192 [2026-10-21 09:15:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0129 [2026-10-21 09:15:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3517 [2026-10-21 09:16:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4812 [2026-10-21 09:16:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3389 [2026-10-21 09:16:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3086 [2026-10-21 09:16:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2324 [2026-10-21 09:16:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2898 [2026-10-21 09:16:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2987 [2026-10-21 09:16:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0654 [2026-10-21 09:16:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6825 [2026-10-21 09:16:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0666 [2026-10-21 09:16:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0747 [2026-10-21 09:16:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7598 [2026-10-21 09:16:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0279 [2026-10-21 09:17:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8959 [2026-10-21 09:17:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0619 [2026-10-21 09:17:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8539 [2026-10-21 09:17:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9846 [2026-10-21 09:17:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7036 [2026-10-21 09:17:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5926 [2026-10-21 09:17:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1659 [2026-10-21 09:17:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0140 [2026-10-21 09:17:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3077 [2026-10-21 09:17:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1086 [2026-10-21 09:17:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6810 [2026-10-21 09:17:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8719 [2026-10-21 09:17:51][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,625 | MFU: 1.8% [2026-10-21 09:17:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1294 [2026-10-21 09:17:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4777 [2026-10-21 09:17:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9567 [2026-10-21 09:17:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2831 [2026-10-21 09:17:56][INFO][training.pretrain_engine] [Ph2|Step 62] loss=5.4314 | ppl=228.47 | aux=0.02507 | lr=3.00e-04 | gnorm=0.516 | tok/s=29,299 | MFU=7.0% | step=143153ms | tokens=0.867B | mem=4.2/32.7GB | cost=$5.10/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:17 (134.0h) | Phase3: SKIPPED | SFT: Oct 27 04:01 (138.7h) | DPO (Final): Oct 27 04:35 (139.3h) [2026-10-21 09:17:56][INFO][training.pretrain_engine] Starting global step 63 (accumulating 32 micro-batches)... [2026-10-21 09:17:56][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 09:18:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0677 [2026-10-21 09:18:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3478 [2026-10-21 09:18:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3656 [2026-10-21 09:18:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5528 [2026-10-21 09:18:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3540 [2026-10-21 09:18:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9958 [2026-10-21 09:18:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0770 [2026-10-21 09:18:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3580 [2026-10-21 09:18:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5909 [2026-10-21 09:18:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5017 [2026-10-21 09:18:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6152 [2026-10-21 09:18:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6414 [2026-10-21 09:19:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9173 [2026-10-21 09:19:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9372 [2026-10-21 09:19:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8012 [2026-10-21 09:19:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6315 [2026-10-21 09:19:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2712 [2026-10-21 09:19:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2604 [2026-10-21 09:19:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5284 [2026-10-21 09:19:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3657 [2026-10-21 09:19:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3849 [2026-10-21 09:19:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2739 [2026-10-21 09:19:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1458 [2026-10-21 09:19:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5624 [2026-10-21 09:20:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8348 [2026-10-21 09:20:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6989 [2026-10-21 09:20:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0362 [2026-10-21 09:20:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2207 [2026-10-21 09:20:14][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,622 | MFU: 1.8% [2026-10-21 09:20:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7815 [2026-10-21 09:20:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7040 [2026-10-21 09:20:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9944 [2026-10-21 09:20:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9915 [2026-10-21 09:20:19][INFO][training.pretrain_engine] [Ph2|Step 63] loss=5.3763 | ppl=216.23 | aux=0.02507 | lr=3.00e-04 | gnorm=0.516 | tok/s=29,301 | MFU=7.0% | step=143143ms | tokens=0.868B | mem=4.2/32.7GB | cost=$5.18/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:18 (134.0h) | Phase3: SKIPPED | SFT: Oct 27 04:02 (138.7h) | DPO (Final): Oct 27 04:37 (139.3h) [2026-10-21 09:20:22][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step63 | Tokens: 0.868B | Spent: $5.18 | Remaining: $994.82 [2026-10-21 09:20:22][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step63 [2026-10-21 09:20:22][INFO][training.pretrain_engine] Starting global step 64 (accumulating 32 micro-batches)... [2026-10-21 09:20:22][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 09:20:22][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 09:20:22][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 09:20:22][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 09:20:22][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step63 to Phase-Technologies/forge-500m-pretrain (path: phase2_step63)... [2026-10-21 09:20:22][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 09:20:32][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 09:20:32][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step63 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 09:20:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0551 [2026-10-21 09:20:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7848 [2026-10-21 09:20:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4181 [2026-10-21 09:20:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5870 [2026-10-21 09:20:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9347 [2026-10-21 09:20:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3840 [2026-10-21 09:20:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6387 [2026-10-21 09:20:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2932 [2026-10-21 09:21:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1704 [2026-10-21 09:21:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2181 [2026-10-21 09:21:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4291 [2026-10-21 09:21:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0713 [2026-10-21 09:21:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3725 [2026-10-21 09:21:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4004 [2026-10-21 09:21:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4839 [2026-10-21 09:21:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1478 [2026-10-21 09:21:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3507 [2026-10-21 09:21:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2195 [2026-10-21 09:21:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1837 [2026-10-21 09:21:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5096 [2026-10-21 09:22:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5970 [2026-10-21 09:22:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4299 [2026-10-21 09:22:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6202 [2026-10-21 09:22:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7136 [2026-10-21 09:22:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9062 [2026-10-21 09:22:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0064 [2026-10-21 09:22:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5417 [2026-10-21 09:22:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9504 [2026-10-21 09:22:39][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,650 | MFU: 1.8% [2026-10-21 09:22:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3402 [2026-10-21 09:22:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0246 [2026-10-21 09:22:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5501 [2026-10-21 09:22:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1287 [2026-10-21 09:22:44][INFO][training.pretrain_engine] [Ph2|Step 64] loss=5.2773 | ppl=195.83 | aux=0.02508 | lr=3.00e-04 | gnorm=0.504 | tok/s=29,306 | MFU=7.0% | step=143121ms | tokens=0.869B | mem=4.2/32.7GB | cost=$5.26/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:19 (133.9h) | Phase3: SKIPPED | SFT: Oct 27 04:03 (138.7h) | DPO (Final): Oct 27 04:37 (139.2h) [2026-10-21 09:22:44][INFO][training.pretrain_engine] Starting global step 65 (accumulating 32 micro-batches)... [2026-10-21 09:22:44][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 09:23:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5502 [2026-10-21 09:23:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3998 [2026-10-21 09:23:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2341 [2026-10-21 09:23:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8787 [2026-10-21 09:23:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5015 [2026-10-21 09:23:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6533 [2026-10-21 09:23:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4038 [2026-10-21 09:23:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6536 [2026-10-21 09:23:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3461 [2026-10-21 09:23:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1561 [2026-10-21 09:23:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8244 [2026-10-21 09:23:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6730 [2026-10-21 09:23:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0490 [2026-10-21 09:23:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4158 [2026-10-21 09:23:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4372 [2026-10-21 09:23:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2828 [2026-10-21 09:24:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1529 [2026-10-21 09:24:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0946 [2026-10-21 09:24:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8700 [2026-10-21 09:24:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3665 [2026-10-21 09:24:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3806 [2026-10-21 09:24:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3298 [2026-10-21 09:24:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7696 [2026-10-21 09:24:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4246 [2026-10-21 09:24:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0845 [2026-10-21 09:24:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7583 [2026-10-21 09:24:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4779 [2026-10-21 09:24:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4017 [2026-10-21 09:25:01][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,650 | MFU: 1.8% [2026-10-21 09:25:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4371 [2026-10-21 09:25:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1792 [2026-10-21 09:25:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9804 [2026-10-21 09:25:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9364 [2026-10-21 09:25:06][INFO][training.pretrain_engine] [Ph2|Step 65] loss=5.3397 | ppl=208.46 | aux=0.02509 | lr=3.00e-04 | gnorm=0.451 | tok/s=29,308 | MFU=7.0% | step=143110ms | tokens=0.870B | mem=4.2/32.7GB | cost=$5.34/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:20 (133.9h) | Phase3: SKIPPED | SFT: Oct 27 04:04 (138.7h) | DPO (Final): Oct 27 04:38 (139.2h) [2026-10-21 09:25:06][INFO][training.pretrain_engine] Starting global step 66 (accumulating 32 micro-batches)... [2026-10-21 09:25:06][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 09:25:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3312 [2026-10-21 09:25:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1655 [2026-10-21 09:25:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0798 [2026-10-21 09:25:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0124 [2026-10-21 09:25:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7955 [2026-10-21 09:25:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6193 [2026-10-21 09:25:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8031 [2026-10-21 09:25:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7923 [2026-10-21 09:25:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1404 [2026-10-21 09:25:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4459 [2026-10-21 09:25:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4954 [2026-10-21 09:25:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1095 [2026-10-21 09:26:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7466 [2026-10-21 09:26:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8091 [2026-10-21 09:26:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6398 [2026-10-21 09:26:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6616 [2026-10-21 09:26:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4916 [2026-10-21 09:26:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3563 [2026-10-21 09:26:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4305 [2026-10-21 09:26:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5423 [2026-10-21 09:26:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6355 [2026-10-21 09:26:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7611 [2026-10-21 09:26:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5455 [2026-10-21 09:26:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6856 [2026-10-21 09:27:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6890 [2026-10-21 09:27:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3800 [2026-10-21 09:27:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3375 [2026-10-21 09:27:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3747 [2026-10-21 09:27:23][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,656 | MFU: 1.8% [2026-10-21 09:27:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1942 [2026-10-21 09:27:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7106 [2026-10-21 09:27:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6425 [2026-10-21 09:27:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7240 [2026-10-21 09:27:28][INFO][training.pretrain_engine] [Ph2|Step 66] loss=5.3722 | ppl=215.34 | aux=0.02507 | lr=3.00e-04 | gnorm=0.459 | tok/s=29,311 | MFU=7.0% | step=143098ms | tokens=0.871B | mem=4.2/32.7GB | cost=$5.42/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:21 (133.9h) | Phase3: SKIPPED | SFT: Oct 27 04:05 (138.6h) | DPO (Final): Oct 27 04:39 (139.2h) [2026-10-21 09:27:28][INFO][training.pretrain_engine] Starting global step 67 (accumulating 32 micro-batches)... [2026-10-21 09:27:28][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 09:27:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.1506 [2026-10-21 09:27:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4436 [2026-10-21 09:27:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6370 [2026-10-21 09:27:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.1998 [2026-10-21 09:28:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6907 [2026-10-21 09:28:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5342 [2026-10-21 09:28:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5295 [2026-10-21 09:28:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2899 [2026-10-21 09:28:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9223 [2026-10-21 09:28:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5671 [2026-10-21 09:28:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0299 [2026-10-21 09:28:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2504 [2026-10-21 09:28:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8941 [2026-10-21 09:28:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8996 [2026-10-21 09:28:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9702 [2026-10-21 09:28:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1652 [2026-10-21 09:28:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4390 [2026-10-21 09:28:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7144 [2026-10-21 09:28:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4755 [2026-10-21 09:28:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4240 [2026-10-21 09:29:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7163 [2026-10-21 09:29:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2775 [2026-10-21 09:29:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3927 [2026-10-21 09:29:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3105 [2026-10-21 09:29:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0337 [2026-10-21 09:29:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4639 [2026-10-21 09:29:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6470 [2026-10-21 09:29:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1246 [2026-10-21 09:29:46][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,628 | MFU: 1.8% [2026-10-21 09:29:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1949 [2026-10-21 09:29:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3079 [2026-10-21 09:29:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3031 [2026-10-21 09:29:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9018 [2026-10-21 09:29:51][INFO][training.pretrain_engine] [Ph2|Step 67] loss=5.4099 | ppl=223.61 | aux=0.02508 | lr=3.00e-04 | gnorm=0.498 | tok/s=29,312 | MFU=7.0% | step=143092ms | tokens=0.872B | mem=4.2/32.7GB | cost=$5.50/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:22 (133.9h) | Phase3: SKIPPED | SFT: Oct 27 04:07 (138.6h) | DPO (Final): Oct 27 04:41 (139.2h) [2026-10-21 09:29:51][INFO][training.pretrain_engine] Starting global step 68 (accumulating 32 micro-batches)... [2026-10-21 09:29:51][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 09:30:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0004 [2026-10-21 09:30:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7992 [2026-10-21 09:30:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6662 [2026-10-21 09:30:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3717 [2026-10-21 09:30:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4068 [2026-10-21 09:30:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1854 [2026-10-21 09:30:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3891 [2026-10-21 09:30:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3523 [2026-10-21 09:30:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8184 [2026-10-21 09:30:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8722 [2026-10-21 09:30:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5347 [2026-10-21 09:30:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3705 [2026-10-21 09:31:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8455 [2026-10-21 09:31:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6085 [2026-10-21 09:31:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4299 [2026-10-21 09:31:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2979 [2026-10-21 09:31:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2581 [2026-10-21 09:31:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7023 [2026-10-21 09:31:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9691 [2026-10-21 09:31:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9596 [2026-10-21 09:31:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3500 [2026-10-21 09:31:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4298 [2026-10-21 09:31:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6593 [2026-10-21 09:31:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5799 [2026-10-21 09:31:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7258 [2026-10-21 09:31:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0631 [2026-10-21 09:31:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3361 [2026-10-21 09:31:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7732 [2026-10-21 09:32:09][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,626 | MFU: 1.8% [2026-10-21 09:32:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9896 [2026-10-21 09:32:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9606 [2026-10-21 09:32:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9182 [2026-10-21 09:32:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9797 [2026-10-21 09:32:14][INFO][training.pretrain_engine] [Ph2|Step 68] loss=5.3311 | ppl=206.67 | aux=0.02507 | lr=3.00e-04 | gnorm=0.443 | tok/s=29,314 | MFU=7.0% | step=143082ms | tokens=0.873B | mem=4.2/32.7GB | cost=$5.59/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:23 (133.9h) | Phase3: SKIPPED | SFT: Oct 27 04:08 (138.6h) | DPO (Final): Oct 27 04:42 (139.2h) [2026-10-21 09:32:14][INFO][training.pretrain_engine] Starting global step 69 (accumulating 32 micro-batches)... [2026-10-21 09:32:14][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 09:32:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3969 [2026-10-21 09:32:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6889 [2026-10-21 09:32:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2310 [2026-10-21 09:32:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2442 [2026-10-21 09:32:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3626 [2026-10-21 09:32:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3689 [2026-10-21 09:32:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1865 [2026-10-21 09:32:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9547 [2026-10-21 09:33:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7272 [2026-10-21 09:33:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8775 [2026-10-21 09:33:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0591 [2026-10-21 09:33:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3593 [2026-10-21 09:33:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8473 [2026-10-21 09:33:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9798 [2026-10-21 09:33:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5766 [2026-10-21 09:33:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6125 [2026-10-21 09:33:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5911 [2026-10-21 09:33:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1456 [2026-10-21 09:33:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7632 [2026-10-21 09:33:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1806 [2026-10-21 09:34:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9478 [2026-10-21 09:34:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2759 [2026-10-21 09:34:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1667 [2026-10-21 09:34:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3532 [2026-10-21 09:34:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5333 [2026-10-21 09:34:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4356 [2026-10-21 09:34:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1929 [2026-10-21 09:34:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2897 [2026-10-21 09:34:31][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,633 | MFU: 1.8% [2026-10-21 09:34:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1939 [2026-10-21 09:34:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6452 [2026-10-21 09:34:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7055 [2026-10-21 09:34:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7086 [2026-10-21 09:34:36][INFO][training.pretrain_engine] [Ph2|Step 69] loss=5.2304 | ppl=186.87 | aux=0.02507 | lr=3.00e-04 | gnorm=0.516 | tok/s=29,316 | MFU=7.0% | step=143074ms | tokens=0.875B | mem=4.2/32.7GB | cost=$5.67/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:25 (133.8h) | Phase3: SKIPPED | SFT: Oct 27 04:09 (138.6h) | DPO (Final): Oct 27 04:43 (139.2h) [2026-10-21 09:34:36][INFO][training.pretrain_engine] Starting global step 70 (accumulating 32 micro-batches)... [2026-10-21 09:34:36][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 09:34:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1094 [2026-10-21 09:34:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3710 [2026-10-21 09:34:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9894 [2026-10-21 09:34:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0640 [2026-10-21 09:35:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5533 [2026-10-21 09:35:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6370 [2026-10-21 09:35:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6458 [2026-10-21 09:35:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2153 [2026-10-21 09:35:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7978 [2026-10-21 09:35:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1051 [2026-10-21 09:35:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3353 [2026-10-21 09:35:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9155 [2026-10-21 09:35:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4794 [2026-10-21 09:35:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9509 [2026-10-21 09:35:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8432 [2026-10-21 09:35:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0390 [2026-10-21 09:36:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5517 [2026-10-21 09:36:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6933 [2026-10-21 09:36:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8883 [2026-10-21 09:36:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5624 [2026-10-21 09:36:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5874 [2026-10-21 09:36:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6057 [2026-10-21 09:36:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3895 [2026-10-21 09:36:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2880 [2026-10-21 09:36:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9799 [2026-10-21 09:36:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9466 [2026-10-21 09:36:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5842 [2026-10-21 09:36:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9717 [2026-10-21 09:36:53][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,637 | MFU: 1.8% [2026-10-21 09:36:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9217 [2026-10-21 09:36:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7903 [2026-10-21 09:36:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6260 [2026-10-21 09:36:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7299 [2026-10-21 09:36:58][INFO][training.pretrain_engine] [Ph2|Step 70] loss=5.3315 | ppl=206.75 | aux=0.02508 | lr=3.00e-04 | gnorm=0.617 | tok/s=29,318 | MFU=7.0% | step=143060ms | tokens=0.876B | mem=4.2/32.7GB | cost=$5.75/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:26 (133.8h) | Phase3: SKIPPED | SFT: Oct 27 04:10 (138.6h) | DPO (Final): Oct 27 04:44 (139.1h) [2026-10-21 09:36:58][INFO][training.pretrain_engine] Starting global step 71 (accumulating 32 micro-batches)... [2026-10-21 09:36:58][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 09:37:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3939 [2026-10-21 09:37:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3373 [2026-10-21 09:37:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4922 [2026-10-21 09:37:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6742 [2026-10-21 09:37:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4655 [2026-10-21 09:37:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3888 [2026-10-21 09:37:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2645 [2026-10-21 09:37:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3412 [2026-10-21 09:37:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2483 [2026-10-21 09:37:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2422 [2026-10-21 09:37:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2288 [2026-10-21 09:37:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2226 [2026-10-21 09:38:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4042 [2026-10-21 09:38:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4180 [2026-10-21 09:38:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5212 [2026-10-21 09:38:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3072 [2026-10-21 09:38:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5991 [2026-10-21 09:38:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3651 [2026-10-21 09:38:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3399 [2026-10-21 09:38:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9978 [2026-10-21 09:38:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7790 [2026-10-21 09:38:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3207 [2026-10-21 09:38:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7430 [2026-10-21 09:38:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5097 [2026-10-21 09:39:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3934 [2026-10-21 09:39:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2280 [2026-10-21 09:39:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1210 [2026-10-21 09:39:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0010 [2026-10-21 09:39:16][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,614 | MFU: 1.8% [2026-10-21 09:39:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4011 [2026-10-21 09:39:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4565 [2026-10-21 09:39:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5496 [2026-10-21 09:39:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6268 [2026-10-21 09:39:21][INFO][training.pretrain_engine] [Ph2|Step 71] loss=5.3506 | ppl=210.74 | aux=0.02507 | lr=3.00e-04 | gnorm=0.957 | tok/s=29,320 | MFU=7.0% | step=143054ms | tokens=0.877B | mem=4.2/32.7GB | cost=$5.83/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:27 (133.8h) | Phase3: SKIPPED | SFT: Oct 27 04:11 (138.5h) | DPO (Final): Oct 27 04:46 (139.1h) [2026-10-21 09:39:21][INFO][training.pretrain_engine] Starting global step 72 (accumulating 32 micro-batches)... [2026-10-21 09:39:21][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 09:39:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2939 [2026-10-21 09:39:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9954 [2026-10-21 09:39:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6015 [2026-10-21 09:39:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1105 [2026-10-21 09:39:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2078 [2026-10-21 09:39:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3176 [2026-10-21 09:39:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1935 [2026-10-21 09:39:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4296 [2026-10-21 09:40:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2779 [2026-10-21 09:40:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7106 [2026-10-21 09:40:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3227 [2026-10-21 09:40:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4865 [2026-10-21 09:40:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1194 [2026-10-21 09:40:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3347 [2026-10-21 09:40:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2918 [2026-10-21 09:40:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2080 [2026-10-21 09:40:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5126 [2026-10-21 09:40:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1372 [2026-10-21 09:40:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3194 [2026-10-21 09:40:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2525 [2026-10-21 09:41:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2717 [2026-10-21 09:41:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5131 [2026-10-21 09:41:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5700 [2026-10-21 09:41:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3289 [2026-10-21 09:41:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2481 [2026-10-21 09:41:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0556 [2026-10-21 09:41:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4145 [2026-10-21 09:41:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1987 [2026-10-21 09:41:38][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,632 | MFU: 1.8% [2026-10-21 09:41:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2635 [2026-10-21 09:41:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6386 [2026-10-21 09:41:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0591 [2026-10-21 09:41:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4558 [2026-10-21 09:41:43][INFO][training.pretrain_engine] [Ph2|Step 72] loss=5.3845 | ppl=218.00 | aux=0.02508 | lr=3.00e-04 | gnorm=1.727 | tok/s=29,321 | MFU=7.0% | step=143047ms | tokens=0.878B | mem=4.2/32.7GB | cost=$5.91/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:29 (133.8h) | Phase3: SKIPPED | SFT: Oct 27 04:13 (138.5h) | DPO (Final): Oct 27 04:47 (139.1h) [2026-10-21 09:41:47][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step72 | Tokens: 0.878B | Spent: $5.91 | Remaining: $994.09 [2026-10-21 09:41:47][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step72 [2026-10-21 09:41:47][INFO][training.pretrain_engine] Starting global step 73 (accumulating 32 micro-batches)... [2026-10-21 09:41:47][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 09:41:47][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 09:41:47][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 09:41:47][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 09:41:47][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step72 to Phase-Technologies/forge-500m-pretrain (path: phase2_step72)... [2026-10-21 09:41:47][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 09:41:58][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 09:41:58][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step72 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 09:42:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0330 [2026-10-21 09:42:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3510 [2026-10-21 09:42:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3719 [2026-10-21 09:42:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7963 [2026-10-21 09:42:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2632 [2026-10-21 09:42:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1083 [2026-10-21 09:42:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9625 [2026-10-21 09:42:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1701 [2026-10-21 09:42:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4824 [2026-10-21 09:42:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0243 [2026-10-21 09:42:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7991 [2026-10-21 09:42:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6512 [2026-10-21 09:42:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3164 [2026-10-21 09:42:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7125 [2026-10-21 09:42:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3936 [2026-10-21 09:42:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2382 [2026-10-21 09:43:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4262 [2026-10-21 09:43:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1607 [2026-10-21 09:43:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6317 [2026-10-21 09:43:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5008 [2026-10-21 09:43:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3505 [2026-10-21 09:43:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0247 [2026-10-21 09:43:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1951 [2026-10-21 09:43:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4232 [2026-10-21 09:43:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5567 [2026-10-21 09:43:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3565 [2026-10-21 09:43:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1060 [2026-10-21 09:43:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4247 [2026-10-21 09:44:04][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,634 | MFU: 1.8% [2026-10-21 09:44:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0742 [2026-10-21 09:44:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1242 [2026-10-21 09:44:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0363 [2026-10-21 09:44:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9065 [2026-10-21 09:44:08][INFO][training.pretrain_engine] [Ph2|Step 73] loss=5.3667 | ppl=214.15 | aux=0.02509 | lr=3.00e-04 | gnorm=0.578 | tok/s=29,325 | MFU=7.0% | step=143030ms | tokens=0.879B | mem=4.2/32.7GB | cost=$6.00/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:29 (133.8h) | Phase3: SKIPPED | SFT: Oct 27 04:14 (138.5h) | DPO (Final): Oct 27 04:48 (139.1h) [2026-10-21 09:44:08][INFO][training.pretrain_engine] Starting global step 74 (accumulating 32 micro-batches)... [2026-10-21 09:44:08][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 09:44:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1647 [2026-10-21 09:44:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1599 [2026-10-21 09:44:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2868 [2026-10-21 09:44:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6388 [2026-10-21 09:44:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1953 [2026-10-21 09:44:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1756 [2026-10-21 09:44:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1002 [2026-10-21 09:44:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2394 [2026-10-21 09:45:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2419 [2026-10-21 09:45:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5931 [2026-10-21 09:45:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2728 [2026-10-21 09:45:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2057 [2026-10-21 09:45:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2092 [2026-10-21 09:45:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6522 [2026-10-21 09:45:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1971 [2026-10-21 09:45:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2606 [2026-10-21 09:45:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3651 [2026-10-21 09:45:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2592 [2026-10-21 09:45:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4337 [2026-10-21 09:45:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2587 [2026-10-21 09:45:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1807 [2026-10-21 09:45:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2167 [2026-10-21 09:45:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4890 [2026-10-21 09:45:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2161 [2026-10-21 09:46:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6773 [2026-10-21 09:46:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4771 [2026-10-21 09:46:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5323 [2026-10-21 09:46:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5439 [2026-10-21 09:46:26][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,620 | MFU: 1.8% [2026-10-21 09:46:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4452 [2026-10-21 09:46:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5192 [2026-10-21 09:46:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3900 [2026-10-21 09:46:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6013 [2026-10-21 09:46:31][INFO][training.pretrain_engine] [Ph2|Step 74] loss=5.4383 | ppl=230.05 | aux=0.02509 | lr=3.00e-04 | gnorm=1.953 | tok/s=29,326 | MFU=7.0% | step=143024ms | tokens=0.880B | mem=4.2/32.7GB | cost=$6.08/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:31 (133.7h) | Phase3: SKIPPED | SFT: Oct 27 04:15 (138.5h) | DPO (Final): Oct 27 04:49 (139.1h) [2026-10-21 09:46:31][INFO][training.pretrain_engine] Starting global step 75 (accumulating 32 micro-batches)... [2026-10-21 09:46:31][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 09:46:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4830 [2026-10-21 09:46:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8362 [2026-10-21 09:46:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6514 [2026-10-21 09:46:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0254 [2026-10-21 09:47:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6230 [2026-10-21 09:47:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5576 [2026-10-21 09:47:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6575 [2026-10-21 09:47:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5146 [2026-10-21 09:47:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2084 [2026-10-21 09:47:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0778 [2026-10-21 09:47:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7428 [2026-10-21 09:47:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9750 [2026-10-21 09:47:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2963 [2026-10-21 09:47:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2818 [2026-10-21 09:47:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4120 [2026-10-21 09:47:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3227 [2026-10-21 09:48:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4426 [2026-10-21 09:48:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4645 [2026-10-21 09:48:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3275 [2026-10-21 09:48:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2615 [2026-10-21 09:48:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2074 [2026-10-21 09:48:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0016 [2026-10-21 09:48:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1848 [2026-10-21 09:48:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8947 [2026-10-21 09:48:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8125 [2026-10-21 09:48:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3958 [2026-10-21 09:48:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9715 [2026-10-21 09:48:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4507 [2026-10-21 09:48:48][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,632 | MFU: 1.8% [2026-10-21 09:48:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3620 [2026-10-21 09:48:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2679 [2026-10-21 09:48:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1887 [2026-10-21 09:48:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3594 [2026-10-21 09:48:54][INFO][training.pretrain_engine] [Ph2|Step 75] loss=5.3541 | ppl=211.47 | aux=0.02508 | lr=3.00e-04 | gnorm=0.805 | tok/s=29,326 | MFU=7.0% | step=143023ms | tokens=0.881B | mem=4.2/32.7GB | cost=$6.16/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:33 (133.7h) | Phase3: SKIPPED | SFT: Oct 27 04:17 (138.5h) | DPO (Final): Oct 27 04:51 (139.0h) [2026-10-21 09:48:54][INFO][training.pretrain_engine] Starting global step 76 (accumulating 32 micro-batches)... [2026-10-21 09:48:54][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 09:49:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5269 [2026-10-21 09:49:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8107 [2026-10-21 09:49:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5651 [2026-10-21 09:49:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9639 [2026-10-21 09:49:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2398 [2026-10-21 09:49:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1923 [2026-10-21 09:49:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3315 [2026-10-21 09:49:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3540 [2026-10-21 09:49:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5819 [2026-10-21 09:49:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0052 [2026-10-21 09:49:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6441 [2026-10-21 09:49:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2690 [2026-10-21 09:50:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2186 [2026-10-21 09:50:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8955 [2026-10-21 09:50:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7403 [2026-10-21 09:50:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4852 [2026-10-21 09:50:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3747 [2026-10-21 09:50:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4347 [2026-10-21 09:50:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6392 [2026-10-21 09:50:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6384 [2026-10-21 09:50:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0246 [2026-10-21 09:50:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2193 [2026-10-21 09:50:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3760 [2026-10-21 09:50:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2302 [2026-10-21 09:50:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6543 [2026-10-21 09:50:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1614 [2026-10-21 09:50:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8068 [2026-10-21 09:50:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6743 [2026-10-21 09:51:11][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,633 | MFU: 1.8% [2026-10-21 09:51:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0820 [2026-10-21 09:51:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7565 [2026-10-21 09:51:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4044 [2026-10-21 09:51:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0996 [2026-10-21 09:51:16][INFO][training.pretrain_engine] [Ph2|Step 76] loss=5.4019 | ppl=221.82 | aux=0.02508 | lr=3.00e-04 | gnorm=1.875 | tok/s=29,328 | MFU=7.0% | step=143014ms | tokens=0.882B | mem=4.2/32.7GB | cost=$6.24/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:34 (133.7h) | Phase3: SKIPPED | SFT: Oct 27 04:18 (138.5h) | DPO (Final): Oct 27 04:52 (139.0h) [2026-10-21 09:51:16][INFO][training.pretrain_engine] Starting global step 77 (accumulating 32 micro-batches)... [2026-10-21 09:51:16][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 09:51:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0644 [2026-10-21 09:51:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8130 [2026-10-21 09:51:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4706 [2026-10-21 09:51:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2348 [2026-10-21 09:51:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1995 [2026-10-21 09:51:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3528 [2026-10-21 09:51:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3865 [2026-10-21 09:51:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4242 [2026-10-21 09:52:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2083 [2026-10-21 09:52:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8861 [2026-10-21 09:52:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6154 [2026-10-21 09:52:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6349 [2026-10-21 09:52:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7201 [2026-10-21 09:52:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0883 [2026-10-21 09:52:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2711 [2026-10-21 09:52:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9821 [2026-10-21 09:52:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2995 [2026-10-21 09:52:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9926 [2026-10-21 09:52:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1871 [2026-10-21 09:52:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6360 [2026-10-21 09:53:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4531 [2026-10-21 09:53:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5085 [2026-10-21 09:53:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3324 [2026-10-21 09:53:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2900 [2026-10-21 09:53:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5312 [2026-10-21 09:53:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9224 [2026-10-21 09:53:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3114 [2026-10-21 09:53:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2754 [2026-10-21 09:53:33][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,640 | MFU: 1.8% [2026-10-21 09:53:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3625 [2026-10-21 09:53:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5969 [2026-10-21 09:53:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3199 [2026-10-21 09:53:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3426 [2026-10-21 09:53:39][INFO][training.pretrain_engine] [Ph2|Step 77] loss=5.3495 | ppl=210.50 | aux=0.02508 | lr=3.00e-04 | gnorm=1.250 | tok/s=29,329 | MFU=7.0% | step=143007ms | tokens=0.883B | mem=4.2/32.7GB | cost=$6.32/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:35 (133.7h) | Phase3: SKIPPED | SFT: Oct 27 04:19 (138.4h) | DPO (Final): Oct 27 04:54 (139.0h) [2026-10-21 09:53:39][INFO][training.pretrain_engine] Starting global step 78 (accumulating 32 micro-batches)... [2026-10-21 09:53:39][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 09:53:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7704 [2026-10-21 09:53:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7291 [2026-10-21 09:53:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9408 [2026-10-21 09:53:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9624 [2026-10-21 09:54:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2872 [2026-10-21 09:54:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1613 [2026-10-21 09:54:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3791 [2026-10-21 09:54:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3680 [2026-10-21 09:54:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4231 [2026-10-21 09:54:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2956 [2026-10-21 09:54:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0160 [2026-10-21 09:54:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4175 [2026-10-21 09:54:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4619 [2026-10-21 09:54:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2842 [2026-10-21 09:54:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9664 [2026-10-21 09:54:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0177 [2026-10-21 09:55:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7553 [2026-10-21 09:55:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0528 [2026-10-21 09:55:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5485 [2026-10-21 09:55:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3536 [2026-10-21 09:55:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4144 [2026-10-21 09:55:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2480 [2026-10-21 09:55:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5783 [2026-10-21 09:55:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6039 [2026-10-21 09:55:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4410 [2026-10-21 09:55:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4919 [2026-10-21 09:55:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5125 [2026-10-21 09:55:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7379 [2026-10-21 09:55:56][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,635 | MFU: 1.8% [2026-10-21 09:56:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8962 [2026-10-21 09:56:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8569 [2026-10-21 09:56:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3423 [2026-10-21 09:56:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7266 [2026-10-21 09:56:01][INFO][training.pretrain_engine] [Ph2|Step 78] loss=5.3707 | ppl=215.01 | aux=0.02508 | lr=3.00e-04 | gnorm=1.336 | tok/s=29,330 | MFU=7.0% | step=143002ms | tokens=0.884B | mem=4.2/32.7GB | cost=$6.40/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:37 (133.7h) | Phase3: SKIPPED | SFT: Oct 27 04:21 (138.4h) | DPO (Final): Oct 27 04:55 (139.0h) [2026-10-21 09:56:01][INFO][training.pretrain_engine] Starting global step 79 (accumulating 32 micro-batches)... [2026-10-21 09:56:01][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 09:56:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9269 [2026-10-21 09:56:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8162 [2026-10-21 09:56:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2088 [2026-10-21 09:56:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9202 [2026-10-21 09:56:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0330 [2026-10-21 09:56:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2766 [2026-10-21 09:56:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2942 [2026-10-21 09:56:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6843 [2026-10-21 09:56:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1491 [2026-10-21 09:56:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4302 [2026-10-21 09:56:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6641 [2026-10-21 09:56:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6292 [2026-10-21 09:57:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5248 [2026-10-21 09:57:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5793 [2026-10-21 09:57:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2903 [2026-10-21 09:57:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4415 [2026-10-21 09:57:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0662 [2026-10-21 09:57:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4597 [2026-10-21 09:57:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7347 [2026-10-21 09:57:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0086 [2026-10-21 09:57:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1755 [2026-10-21 09:57:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8924 [2026-10-21 09:57:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5857 [2026-10-21 09:57:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8439 [2026-10-21 09:58:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3333 [2026-10-21 09:58:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2915 [2026-10-21 09:58:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4934 [2026-10-21 09:58:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1948 [2026-10-21 09:58:19][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,642 | MFU: 1.8% [2026-10-21 09:58:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0854 [2026-10-21 09:58:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9809 [2026-10-21 09:58:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4692 [2026-10-21 09:58:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5116 [2026-10-21 09:58:24][INFO][training.pretrain_engine] [Ph2|Step 79] loss=5.3173 | ppl=203.83 | aux=0.02510 | lr=3.00e-04 | gnorm=1.281 | tok/s=29,332 | MFU=7.0% | step=142995ms | tokens=0.885B | mem=4.2/32.7GB | cost=$6.48/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:38 (133.7h) | Phase3: SKIPPED | SFT: Oct 27 04:22 (138.4h) | DPO (Final): Oct 27 04:56 (139.0h) [2026-10-21 09:58:24][INFO][training.pretrain_engine] Starting global step 80 (accumulating 32 micro-batches)... [2026-10-21 09:58:24][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 09:58:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2196 [2026-10-21 09:58:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8732 [2026-10-21 09:58:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1625 [2026-10-21 09:58:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5662 [2026-10-21 09:58:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5922 [2026-10-21 09:58:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3515 [2026-10-21 09:58:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2300 [2026-10-21 09:58:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0279 [2026-10-21 09:59:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3524 [2026-10-21 09:59:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3299 [2026-10-21 09:59:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7179 [2026-10-21 09:59:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0939 [2026-10-21 09:59:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1681 [2026-10-21 09:59:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3465 [2026-10-21 09:59:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3339 [2026-10-21 09:59:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3563 [2026-10-21 09:59:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1029 [2026-10-21 09:59:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2095 [2026-10-21 09:59:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3760 [2026-10-21 09:59:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9630 [2026-10-21 10:00:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4059 [2026-10-21 10:00:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1412 [2026-10-21 10:00:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1205 [2026-10-21 10:00:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4873 [2026-10-21 10:00:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4207 [2026-10-21 10:00:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3423 [2026-10-21 10:00:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4773 [2026-10-21 10:00:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2917 [2026-10-21 10:00:41][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,636 | MFU: 1.8% [2026-10-21 10:00:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0074 [2026-10-21 10:00:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3933 [2026-10-21 10:00:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2050 [2026-10-21 10:00:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2335 [2026-10-21 10:00:46][INFO][training.pretrain_engine] [Ph2|Step 80] loss=5.3038 | ppl=201.11 | aux=0.02509 | lr=3.00e-04 | gnorm=1.062 | tok/s=29,333 | MFU=7.0% | step=142987ms | tokens=0.886B | mem=4.2/32.7GB | cost=$6.57/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:40 (133.7h) | Phase3: SKIPPED | SFT: Oct 27 04:24 (138.4h) | DPO (Final): Oct 27 04:58 (139.0h) [2026-10-21 10:00:46][INFO][training.pretrain_engine] Starting global step 81 (accumulating 32 micro-batches)... [2026-10-21 10:00:46][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 10:01:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3145 [2026-10-21 10:01:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1779 [2026-10-21 10:01:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4841 [2026-10-21 10:01:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8092 [2026-10-21 10:01:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0337 [2026-10-21 10:01:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7810 [2026-10-21 10:01:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4886 [2026-10-21 10:01:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9372 [2026-10-21 10:01:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4245 [2026-10-21 10:01:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3824 [2026-10-21 10:01:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3397 [2026-10-21 10:01:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9267 [2026-10-21 10:01:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6006 [2026-10-21 10:01:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8433 [2026-10-21 10:01:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2702 [2026-10-21 10:01:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9423 [2026-10-21 10:02:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3268 [2026-10-21 10:02:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7413 [2026-10-21 10:02:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1980 [2026-10-21 10:02:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2171 [2026-10-21 10:02:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7660 [2026-10-21 10:02:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1747 [2026-10-21 10:02:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0188 [2026-10-21 10:02:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0709 [2026-10-21 10:02:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1686 [2026-10-21 10:02:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3737 [2026-10-21 10:02:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6302 [2026-10-21 10:02:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2773 [2026-10-21 10:03:04][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,633 | MFU: 1.8% [2026-10-21 10:03:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7639 [2026-10-21 10:03:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1548 [2026-10-21 10:03:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1195 [2026-10-21 10:03:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5256 [2026-10-21 10:03:09][INFO][training.pretrain_engine] [Ph2|Step 81] loss=5.3283 | ppl=206.08 | aux=0.02509 | lr=3.00e-04 | gnorm=1.023 | tok/s=29,335 | MFU=7.0% | step=142982ms | tokens=0.887B | mem=4.2/32.7GB | cost=$6.65/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:41 (133.6h) | Phase3: SKIPPED | SFT: Oct 27 04:25 (138.4h) | DPO (Final): Oct 27 04:59 (138.9h) [2026-10-21 10:03:12][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step81 | Tokens: 0.887B | Spent: $6.65 | Remaining: $993.35 [2026-10-21 10:03:12][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step81 [2026-10-21 10:03:12][INFO][training.pretrain_engine] Starting global step 82 (accumulating 32 micro-batches)... [2026-10-21 10:03:12][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 10:03:12][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 10:03:12][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 10:03:12][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 10:03:12][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step81 to Phase-Technologies/forge-500m-pretrain (path: phase2_step81)... [2026-10-21 10:03:12][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 10:03:22][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 10:03:22][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step81 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 10:03:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4097 [2026-10-21 10:03:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9751 [2026-10-21 10:03:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2325 [2026-10-21 10:03:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0567 [2026-10-21 10:03:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7868 [2026-10-21 10:03:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4496 [2026-10-21 10:03:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3212 [2026-10-21 10:03:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5120 [2026-10-21 10:04:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1996 [2026-10-21 10:04:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6351 [2026-10-21 10:04:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8331 [2026-10-21 10:04:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3822 [2026-10-21 10:04:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4097 [2026-10-21 10:04:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2475 [2026-10-21 10:04:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4496 [2026-10-21 10:04:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2643 [2026-10-21 10:04:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5029 [2026-10-21 10:04:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8437 [2026-10-21 10:04:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8801 [2026-10-21 10:04:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0233 [2026-10-21 10:04:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1163 [2026-10-21 10:04:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3415 [2026-10-21 10:04:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9383 [2026-10-21 10:04:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5988 [2026-10-21 10:05:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2171 [2026-10-21 10:05:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6405 [2026-10-21 10:05:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2348 [2026-10-21 10:05:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1217 [2026-10-21 10:05:29][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,631 | MFU: 1.8% [2026-10-21 10:05:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2553 [2026-10-21 10:05:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5941 [2026-10-21 10:05:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6655 [2026-10-21 10:05:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5130 [2026-10-21 10:05:34][INFO][training.pretrain_engine] [Ph2|Step 82] loss=5.3202 | ppl=204.43 | aux=0.02511 | lr=3.00e-04 | gnorm=0.867 | tok/s=29,337 | MFU=7.0% | step=142968ms | tokens=0.888B | mem=4.2/32.7GB | cost=$6.73/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:42 (133.6h) | Phase3: SKIPPED | SFT: Oct 27 04:26 (138.4h) | DPO (Final): Oct 27 05:00 (138.9h) [2026-10-21 10:05:34][INFO][training.pretrain_engine] Starting global step 83 (accumulating 32 micro-batches)... [2026-10-21 10:05:34][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 10:05:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5973 [2026-10-21 10:05:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7549 [2026-10-21 10:05:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3453 [2026-10-21 10:05:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0899 [2026-10-21 10:06:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4557 [2026-10-21 10:06:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3547 [2026-10-21 10:06:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4121 [2026-10-21 10:06:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3102 [2026-10-21 10:06:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3480 [2026-10-21 10:06:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4408 [2026-10-21 10:06:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2229 [2026-10-21 10:06:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9964 [2026-10-21 10:06:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2044 [2026-10-21 10:06:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7848 [2026-10-21 10:06:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0478 [2026-10-21 10:06:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0745 [2026-10-21 10:07:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6511 [2026-10-21 10:07:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9196 [2026-10-21 10:07:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.0750 [2026-10-21 10:07:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3007 [2026-10-21 10:07:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2222 [2026-10-21 10:07:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4741 [2026-10-21 10:07:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5277 [2026-10-21 10:07:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4639 [2026-10-21 10:07:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2342 [2026-10-21 10:07:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4539 [2026-10-21 10:07:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5435 [2026-10-21 10:07:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0438 [2026-10-21 10:07:51][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,637 | MFU: 1.8% [2026-10-21 10:07:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1140 [2026-10-21 10:07:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0187 [2026-10-21 10:07:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8795 [2026-10-21 10:07:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0425 [2026-10-21 10:07:56][INFO][training.pretrain_engine] [Ph2|Step 83] loss=5.4247 | ppl=226.94 | aux=0.02508 | lr=3.00e-04 | gnorm=0.820 | tok/s=29,339 | MFU=7.0% | step=142961ms | tokens=0.889B | mem=4.2/32.7GB | cost=$6.81/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:43 (133.6h) | Phase3: SKIPPED | SFT: Oct 27 04:28 (138.3h) | DPO (Final): Oct 27 05:02 (138.9h) [2026-10-21 10:07:56][INFO][training.pretrain_engine] Starting global step 84 (accumulating 32 micro-batches)... [2026-10-21 10:07:56][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 10:08:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8561 [2026-10-21 10:08:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6054 [2026-10-21 10:08:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1563 [2026-10-21 10:08:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5566 [2026-10-21 10:08:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6090 [2026-10-21 10:08:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4861 [2026-10-21 10:08:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3332 [2026-10-21 10:08:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9863 [2026-10-21 10:08:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4712 [2026-10-21 10:08:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4739 [2026-10-21 10:08:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3602 [2026-10-21 10:08:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7237 [2026-10-21 10:09:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2497 [2026-10-21 10:09:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4594 [2026-10-21 10:09:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1589 [2026-10-21 10:09:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1125 [2026-10-21 10:09:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9651 [2026-10-21 10:09:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6306 [2026-10-21 10:09:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9071 [2026-10-21 10:09:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3070 [2026-10-21 10:09:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4178 [2026-10-21 10:09:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1165 [2026-10-21 10:09:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7131 [2026-10-21 10:09:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5176 [2026-10-21 10:10:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3536 [2026-10-21 10:10:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3424 [2026-10-21 10:10:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2241 [2026-10-21 10:10:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1740 [2026-10-21 10:10:14][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,619 | MFU: 1.8% [2026-10-21 10:10:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4827 [2026-10-21 10:10:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1773 [2026-10-21 10:10:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0730 [2026-10-21 10:10:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3743 [2026-10-21 10:10:19][INFO][training.pretrain_engine] [Ph2|Step 84] loss=5.3694 | ppl=214.73 | aux=0.02508 | lr=3.00e-04 | gnorm=0.770 | tok/s=29,340 | MFU=7.0% | step=142957ms | tokens=0.890B | mem=4.2/32.7GB | cost=$6.89/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:45 (133.6h) | Phase3: SKIPPED | SFT: Oct 27 04:29 (138.3h) | DPO (Final): Oct 27 05:03 (138.9h) [2026-10-21 10:10:19][INFO][training.pretrain_engine] Starting global step 85 (accumulating 32 micro-batches)... [2026-10-21 10:10:19][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 10:10:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2881 [2026-10-21 10:10:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5698 [2026-10-21 10:10:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5249 [2026-10-21 10:10:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0318 [2026-10-21 10:10:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4507 [2026-10-21 10:10:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2418 [2026-10-21 10:10:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2275 [2026-10-21 10:10:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2830 [2026-10-21 10:11:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4710 [2026-10-21 10:11:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8001 [2026-10-21 10:11:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9331 [2026-10-21 10:11:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6709 [2026-10-21 10:11:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2414 [2026-10-21 10:11:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9648 [2026-10-21 10:11:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6213 [2026-10-21 10:11:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0740 [2026-10-21 10:11:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8586 [2026-10-21 10:11:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3004 [2026-10-21 10:11:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6404 [2026-10-21 10:11:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4742 [2026-10-21 10:12:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3408 [2026-10-21 10:12:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5625 [2026-10-21 10:12:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4954 [2026-10-21 10:12:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3122 [2026-10-21 10:12:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4382 [2026-10-21 10:12:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2040 [2026-10-21 10:12:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7451 [2026-10-21 10:12:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.2276 [2026-10-21 10:12:37][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,616 | MFU: 1.8% [2026-10-21 10:12:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2244 [2026-10-21 10:12:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4196 [2026-10-21 10:12:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4318 [2026-10-21 10:12:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2728 [2026-10-21 10:12:42][INFO][training.pretrain_engine] [Ph2|Step 85] loss=5.4023 | ppl=221.91 | aux=0.02508 | lr=3.00e-04 | gnorm=0.789 | tok/s=29,339 | MFU=7.0% | step=142959ms | tokens=0.891B | mem=4.2/32.7GB | cost=$6.98/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:47 (133.6h) | Phase3: SKIPPED | SFT: Oct 27 04:31 (138.3h) | DPO (Final): Oct 27 05:05 (138.9h) [2026-10-21 10:12:42][INFO][training.pretrain_engine] Starting global step 86 (accumulating 32 micro-batches)... [2026-10-21 10:12:42][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 10:13:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2220 [2026-10-21 10:13:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2822 [2026-10-21 10:13:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6093 [2026-10-21 10:13:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4159 [2026-10-21 10:13:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8479 [2026-10-21 10:13:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8305 [2026-10-21 10:13:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0496 [2026-10-21 10:13:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5528 [2026-10-21 10:13:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2284 [2026-10-21 10:13:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4077 [2026-10-21 10:13:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.0150 [2026-10-21 10:13:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6966 [2026-10-21 10:13:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1118 [2026-10-21 10:13:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0832 [2026-10-21 10:13:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2563 [2026-10-21 10:13:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9618 [2026-10-21 10:14:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6522 [2026-10-21 10:14:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7894 [2026-10-21 10:14:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0693 [2026-10-21 10:14:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6960 [2026-10-21 10:14:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1524 [2026-10-21 10:14:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3990 [2026-10-21 10:14:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2225 [2026-10-21 10:14:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0465 [2026-10-21 10:14:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6911 [2026-10-21 10:14:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6486 [2026-10-21 10:14:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5949 [2026-10-21 10:14:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5087 [2026-10-21 10:15:00][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,601 | MFU: 1.8% [2026-10-21 10:15:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9593 [2026-10-21 10:15:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1485 [2026-10-21 10:15:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7617 [2026-10-21 10:15:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2619 [2026-10-21 10:15:05][INFO][training.pretrain_engine] [Ph2|Step 86] loss=5.2097 | ppl=183.04 | aux=0.02509 | lr=3.00e-04 | gnorm=0.672 | tok/s=29,338 | MFU=7.0% | step=142964ms | tokens=0.892B | mem=4.2/32.7GB | cost=$7.06/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:49 (133.6h) | Phase3: SKIPPED | SFT: Oct 27 04:33 (138.3h) | DPO (Final): Oct 27 05:07 (138.9h) [2026-10-21 10:15:05][INFO][training.pretrain_engine] Starting global step 87 (accumulating 32 micro-batches)... [2026-10-21 10:15:05][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 10:15:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3858 [2026-10-21 10:15:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7191 [2026-10-21 10:15:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5667 [2026-10-21 10:15:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2677 [2026-10-21 10:15:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2763 [2026-10-21 10:15:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0624 [2026-10-21 10:15:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1618 [2026-10-21 10:15:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2791 [2026-10-21 10:15:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5907 [2026-10-21 10:15:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5193 [2026-10-21 10:15:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4750 [2026-10-21 10:15:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4689 [2026-10-21 10:16:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0779 [2026-10-21 10:16:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5291 [2026-10-21 10:16:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4440 [2026-10-21 10:16:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4155 [2026-10-21 10:16:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7801 [2026-10-21 10:16:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5467 [2026-10-21 10:16:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3856 [2026-10-21 10:16:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9446 [2026-10-21 10:16:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9490 [2026-10-21 10:16:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5272 [2026-10-21 10:16:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8444 [2026-10-21 10:16:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9608 [2026-10-21 10:17:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1413 [2026-10-21 10:17:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0693 [2026-10-21 10:17:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3512 [2026-10-21 10:17:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9675 [2026-10-21 10:17:23][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,596 | MFU: 1.8% [2026-10-21 10:17:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5857 [2026-10-21 10:17:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3012 [2026-10-21 10:17:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0035 [2026-10-21 10:17:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8234 [2026-10-21 10:17:29][INFO][training.pretrain_engine] [Ph2|Step 87] loss=5.3629 | ppl=213.35 | aux=0.02508 | lr=3.00e-04 | gnorm=0.609 | tok/s=29,337 | MFU=7.0% | step=142969ms | tokens=0.893B | mem=4.2/32.7GB | cost=$7.14/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:51 (133.6h) | Phase3: SKIPPED | SFT: Oct 27 04:35 (138.3h) | DPO (Final): Oct 27 05:09 (138.9h) [2026-10-21 10:17:29][INFO][training.pretrain_engine] Starting global step 88 (accumulating 32 micro-batches)... [2026-10-21 10:17:29][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 10:17:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8390 [2026-10-21 10:17:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2253 [2026-10-21 10:17:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4214 [2026-10-21 10:17:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5991 [2026-10-21 10:18:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5796 [2026-10-21 10:18:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8092 [2026-10-21 10:18:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6778 [2026-10-21 10:18:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5345 [2026-10-21 10:18:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5402 [2026-10-21 10:18:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3489 [2026-10-21 10:18:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4331 [2026-10-21 10:18:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3198 [2026-10-21 10:18:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5460 [2026-10-21 10:18:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7865 [2026-10-21 10:18:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1899 [2026-10-21 10:18:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2759 [2026-10-21 10:18:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1959 [2026-10-21 10:18:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3524 [2026-10-21 10:18:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5422 [2026-10-21 10:18:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2618 [2026-10-21 10:19:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2762 [2026-10-21 10:19:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1540 [2026-10-21 10:19:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3880 [2026-10-21 10:19:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3129 [2026-10-21 10:19:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7498 [2026-10-21 10:19:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5127 [2026-10-21 10:19:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1232 [2026-10-21 10:19:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1089 [2026-10-21 10:19:47][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,588 | MFU: 1.8% [2026-10-21 10:19:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9178 [2026-10-21 10:19:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3543 [2026-10-21 10:19:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5140 [2026-10-21 10:19:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3484 [2026-10-21 10:19:52][INFO][training.pretrain_engine] [Ph2|Step 88] loss=5.3819 | ppl=217.43 | aux=0.02507 | lr=3.00e-04 | gnorm=0.680 | tok/s=29,336 | MFU=7.0% | step=142974ms | tokens=0.894B | mem=4.2/32.7GB | cost=$7.22/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:53 (133.6h) | Phase3: SKIPPED | SFT: Oct 27 04:37 (138.3h) | DPO (Final): Oct 27 05:11 (138.9h) [2026-10-21 10:19:52][INFO][training.pretrain_engine] Starting global step 89 (accumulating 32 micro-batches)... [2026-10-21 10:19:52][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 10:20:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2857 [2026-10-21 10:20:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5757 [2026-10-21 10:20:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4845 [2026-10-21 10:20:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0536 [2026-10-21 10:20:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7659 [2026-10-21 10:20:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2757 [2026-10-21 10:20:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8674 [2026-10-21 10:20:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0624 [2026-10-21 10:20:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5986 [2026-10-21 10:20:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3598 [2026-10-21 10:20:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3244 [2026-10-21 10:20:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4690 [2026-10-21 10:21:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4006 [2026-10-21 10:21:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2358 [2026-10-21 10:21:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1545 [2026-10-21 10:21:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7388 [2026-10-21 10:21:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1704 [2026-10-21 10:21:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3212 [2026-10-21 10:21:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2964 [2026-10-21 10:21:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1671 [2026-10-21 10:21:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1728 [2026-10-21 10:21:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4480 [2026-10-21 10:21:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1584 [2026-10-21 10:21:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9253 [2026-10-21 10:21:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4216 [2026-10-21 10:21:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5298 [2026-10-21 10:21:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6485 [2026-10-21 10:21:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4016 [2026-10-21 10:22:10][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,606 | MFU: 1.8% [2026-10-21 10:22:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2605 [2026-10-21 10:22:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8664 [2026-10-21 10:22:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8858 [2026-10-21 10:22:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3353 [2026-10-21 10:22:15][INFO][training.pretrain_engine] [Ph2|Step 89] loss=5.3099 | ppl=202.33 | aux=0.02508 | lr=3.00e-04 | gnorm=0.656 | tok/s=29,336 | MFU=7.0% | step=142977ms | tokens=0.895B | mem=4.2/32.7GB | cost=$7.30/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:55 (133.6h) | Phase3: SKIPPED | SFT: Oct 27 04:39 (138.3h) | DPO (Final): Oct 27 05:13 (138.9h) [2026-10-21 10:22:15][INFO][training.pretrain_engine] Starting global step 90 (accumulating 32 micro-batches)... [2026-10-21 10:22:15][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 10:22:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7784 [2026-10-21 10:22:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2099 [2026-10-21 10:22:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8856 [2026-10-21 10:22:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.1260 [2026-10-21 10:22:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6918 [2026-10-21 10:22:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4785 [2026-10-21 10:22:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2406 [2026-10-21 10:22:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1521 [2026-10-21 10:23:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3590 [2026-10-21 10:23:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6968 [2026-10-21 10:23:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2340 [2026-10-21 10:23:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2180 [2026-10-21 10:23:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1944 [2026-10-21 10:23:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4709 [2026-10-21 10:23:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1518 [2026-10-21 10:23:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4312 [2026-10-21 10:23:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3898 [2026-10-21 10:23:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4848 [2026-10-21 10:23:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4025 [2026-10-21 10:23:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4177 [2026-10-21 10:24:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6548 [2026-10-21 10:24:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2594 [2026-10-21 10:24:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4059 [2026-10-21 10:24:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2989 [2026-10-21 10:24:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1405 [2026-10-21 10:24:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1974 [2026-10-21 10:24:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1687 [2026-10-21 10:24:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2707 [2026-10-21 10:24:33][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,615 | MFU: 1.8% [2026-10-21 10:24:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9230 [2026-10-21 10:24:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5654 [2026-10-21 10:24:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7218 [2026-10-21 10:24:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6299 [2026-10-21 10:24:39][INFO][training.pretrain_engine] [Ph2|Step 90] loss=5.3807 | ppl=217.17 | aux=0.02507 | lr=3.00e-04 | gnorm=0.523 | tok/s=29,334 | MFU=7.0% | step=142983ms | tokens=0.897B | mem=4.2/32.7GB | cost=$7.39/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:57 (133.6h) | Phase3: SKIPPED | SFT: Oct 27 04:41 (138.3h) | DPO (Final): Oct 27 05:15 (138.9h) [2026-10-21 10:24:42][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step90 | Tokens: 0.897B | Spent: $7.39 | Remaining: $992.61 [2026-10-21 10:24:42][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step90 [2026-10-21 10:24:42][INFO][training.pretrain_engine] Starting global step 91 (accumulating 32 micro-batches)... [2026-10-21 10:24:42][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 10:24:42][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 10:24:42][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 10:24:42][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 10:24:42][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step90 to Phase-Technologies/forge-500m-pretrain (path: phase2_step90)... [2026-10-21 10:24:43][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 10:24:52][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 10:24:52][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step90 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 10:24:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2906 [2026-10-21 10:24:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4538 [2026-10-21 10:24:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0563 [2026-10-21 10:25:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3679 [2026-10-21 10:25:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7635 [2026-10-21 10:25:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8528 [2026-10-21 10:25:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2924 [2026-10-21 10:25:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2499 [2026-10-21 10:25:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1930 [2026-10-21 10:25:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8879 [2026-10-21 10:25:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0481 [2026-10-21 10:25:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1402 [2026-10-21 10:25:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3956 [2026-10-21 10:25:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3517 [2026-10-21 10:25:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3424 [2026-10-21 10:25:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3573 [2026-10-21 10:26:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4094 [2026-10-21 10:26:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3438 [2026-10-21 10:26:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5670 [2026-10-21 10:26:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6143 [2026-10-21 10:26:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2839 [2026-10-21 10:26:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0813 [2026-10-21 10:26:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6329 [2026-10-21 10:26:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1592 [2026-10-21 10:26:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6809 [2026-10-21 10:26:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9418 [2026-10-21 10:26:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7994 [2026-10-21 10:26:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7137 [2026-10-21 10:27:00][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,609 | MFU: 1.8% [2026-10-21 10:27:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3134 [2026-10-21 10:27:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5097 [2026-10-21 10:27:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7594 [2026-10-21 10:27:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2590 [2026-10-21 10:27:05][INFO][training.pretrain_engine] [Ph2|Step 91] loss=5.3475 | ppl=210.08 | aux=0.02507 | lr=2.99e-04 | gnorm=0.668 | tok/s=29,336 | MFU=7.0% | step=142975ms | tokens=0.898B | mem=4.2/32.7GB | cost=$7.47/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:59 (133.5h) | Phase3: SKIPPED | SFT: Oct 27 04:43 (138.3h) | DPO (Final): Oct 27 05:17 (138.8h) [2026-10-21 10:27:05][INFO][training.pretrain_engine] Starting global step 92 (accumulating 32 micro-batches)... [2026-10-21 10:27:05][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 10:27:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3995 [2026-10-21 10:27:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1523 [2026-10-21 10:27:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6980 [2026-10-21 10:27:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4064 [2026-10-21 10:27:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2007 [2026-10-21 10:27:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3518 [2026-10-21 10:27:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0943 [2026-10-21 10:27:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3054 [2026-10-21 10:27:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3174 [2026-10-21 10:27:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9770 [2026-10-21 10:27:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1805 [2026-10-21 10:27:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0553 [2026-10-21 10:28:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6081 [2026-10-21 10:28:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3454 [2026-10-21 10:28:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4743 [2026-10-21 10:28:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3360 [2026-10-21 10:28:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4635 [2026-10-21 10:28:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1946 [2026-10-21 10:28:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7618 [2026-10-21 10:28:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8921 [2026-10-21 10:28:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5020 [2026-10-21 10:28:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4195 [2026-10-21 10:28:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4304 [2026-10-21 10:28:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2719 [2026-10-21 10:29:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3356 [2026-10-21 10:29:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4057 [2026-10-21 10:29:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6722 [2026-10-21 10:29:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0300 [2026-10-21 10:29:23][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,602 | MFU: 1.8% [2026-10-21 10:29:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2846 [2026-10-21 10:29:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6400 [2026-10-21 10:29:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0467 [2026-10-21 10:29:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4616 [2026-10-21 10:29:28][INFO][training.pretrain_engine] [Ph2|Step 92] loss=5.3425 | ppl=209.03 | aux=0.02507 | lr=2.99e-04 | gnorm=0.559 | tok/s=29,335 | MFU=7.0% | step=142982ms | tokens=0.899B | mem=4.2/32.7GB | cost=$7.55/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 00:01 (133.5h) | Phase3: SKIPPED | SFT: Oct 27 04:45 (138.3h) | DPO (Final): Oct 27 05:19 (138.8h) [2026-10-21 10:29:28][INFO][training.pretrain_engine] Starting global step 93 (accumulating 32 micro-batches)... [2026-10-21 10:29:28][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 10:29:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3085 [2026-10-21 10:29:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1438 [2026-10-21 10:29:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8446 [2026-10-21 10:29:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2709 [2026-10-21 10:30:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4681 [2026-10-21 10:30:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5077 [2026-10-21 10:30:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3815 [2026-10-21 10:30:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6923 [2026-10-21 10:30:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1792 [2026-10-21 10:30:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2811 [2026-10-21 10:30:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0855 [2026-10-21 10:30:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4601 [2026-10-21 10:30:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2218 [2026-10-21 10:30:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8694 [2026-10-21 10:30:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1876 [2026-10-21 10:30:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1162 [2026-10-21 10:30:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2985 [2026-10-21 10:30:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8874 [2026-10-21 10:30:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2681 [2026-10-21 10:30:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1484 [2026-10-21 10:31:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3286 [2026-10-21 10:31:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5710 [2026-10-21 10:31:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7096 [2026-10-21 10:31:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0091 [2026-10-21 10:31:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5644 [2026-10-21 10:31:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4962 [2026-10-21 10:31:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0061 [2026-10-21 10:31:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3503 [2026-10-21 10:31:46][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,607 | MFU: 1.8% [2026-10-21 10:31:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7710 [2026-10-21 10:31:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9351 [2026-10-21 10:31:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9449 [2026-10-21 10:31:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0919 [2026-10-21 10:31:52][INFO][training.pretrain_engine] [Ph2|Step 93] loss=5.3618 | ppl=213.11 | aux=0.02508 | lr=2.99e-04 | gnorm=0.625 | tok/s=29,333 | MFU=7.0% | step=142987ms | tokens=0.900B | mem=4.2/32.7GB | cost=$7.63/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 00:03 (133.5h) | Phase3: SKIPPED | SFT: Oct 27 04:47 (138.3h) | DPO (Final): Oct 27 05:21 (138.8h) [2026-10-21 10:31:52][INFO][training.pretrain_engine] Starting global step 94 (accumulating 32 micro-batches)... [2026-10-21 10:31:52][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 10:32:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4870 [2026-10-21 10:32:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1829 [2026-10-21 10:32:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5083 [2026-10-21 10:32:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2169 [2026-10-21 10:32:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4338 [2026-10-21 10:32:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5151 [2026-10-21 10:32:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6376 [2026-10-21 10:32:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3308 [2026-10-21 10:32:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9617 [2026-10-21 10:32:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6824 [2026-10-21 10:32:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0105 [2026-10-21 10:32:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5202 [2026-10-21 10:33:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2871 [2026-10-21 10:33:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6911 [2026-10-21 10:33:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3512 [2026-10-21 10:33:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3008 [2026-10-21 10:33:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7271 [2026-10-21 10:33:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4797 [2026-10-21 10:33:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0533 [2026-10-21 10:33:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3531 [2026-10-21 10:33:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3117 [2026-10-21 10:33:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5471 [2026-10-21 10:33:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3839 [2026-10-21 10:33:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6139 [2026-10-21 10:33:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9610 [2026-10-21 10:33:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5963 [2026-10-21 10:33:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2079 [2026-10-21 10:33:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4456 [2026-10-21 10:34:09][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,621 | MFU: 1.8% [2026-10-21 10:34:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0602 [2026-10-21 10:34:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0980 [2026-10-21 10:34:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2472 [2026-10-21 10:34:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0448 [2026-10-21 10:34:15][INFO][training.pretrain_engine] [Ph2|Step 94] loss=5.3682 | ppl=214.48 | aux=0.02506 | lr=2.99e-04 | gnorm=0.504 | tok/s=29,332 | MFU=7.0% | step=142992ms | tokens=0.901B | mem=4.2/32.7GB | cost=$7.72/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 00:05 (133.5h) | Phase3: SKIPPED | SFT: Oct 27 04:49 (138.3h) | DPO (Final): Oct 27 05:23 (138.8h) [2026-10-21 10:34:15][INFO][training.pretrain_engine] Starting global step 95 (accumulating 32 micro-batches)... [2026-10-21 10:34:15][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 10:34:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1229 [2026-10-21 10:34:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7728 [2026-10-21 10:34:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0490 [2026-10-21 10:34:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0619 [2026-10-21 10:34:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7002 [2026-10-21 10:34:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6017 [2026-10-21 10:34:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6149 [2026-10-21 10:34:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4362 [2026-10-21 10:35:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8097 [2026-10-21 10:35:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5159 [2026-10-21 10:35:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5777 [2026-10-21 10:35:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5317 [2026-10-21 10:35:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1518 [2026-10-21 10:35:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2137 [2026-10-21 10:35:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2104 [2026-10-21 10:35:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8281 [2026-10-21 10:35:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4019 [2026-10-21 10:35:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8029 [2026-10-21 10:35:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2480 [2026-10-21 10:35:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4911 [2026-10-21 10:36:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5098 [2026-10-21 10:36:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4121 [2026-10-21 10:36:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0596 [2026-10-21 10:36:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1625 [2026-10-21 10:36:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9093 [2026-10-21 10:36:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8784 [2026-10-21 10:36:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3144 [2026-10-21 10:36:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1499 [2026-10-21 10:36:33][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,608 | MFU: 1.8% [2026-10-21 10:36:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2440 [2026-10-21 10:36:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6901 [2026-10-21 10:36:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4863 [2026-10-21 10:36:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6904 [2026-10-21 10:36:38][INFO][training.pretrain_engine] [Ph2|Step 95] loss=5.3458 | ppl=209.72 | aux=0.02506 | lr=2.99e-04 | gnorm=0.562 | tok/s=29,332 | MFU=7.0% | step=142992ms | tokens=0.902B | mem=4.2/32.7GB | cost=$7.80/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 00:07 (133.5h) | Phase3: SKIPPED | SFT: Oct 27 04:51 (138.2h) | DPO (Final): Oct 27 05:25 (138.8h) [2026-10-21 10:36:38][INFO][training.pretrain_engine] Starting global step 96 (accumulating 32 micro-batches)... [2026-10-21 10:36:38][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 10:36:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5234 [2026-10-21 10:36:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4248 [2026-10-21 10:36:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2279 [2026-10-21 10:36:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4457 [2026-10-21 10:37:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3575 [2026-10-21 10:37:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2574 [2026-10-21 10:37:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6182 [2026-10-21 10:37:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6561 [2026-10-21 10:37:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8995 [2026-10-21 10:37:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7176 [2026-10-21 10:37:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3634 [2026-10-21 10:37:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3858 [2026-10-21 10:37:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0473 [2026-10-21 10:37:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7594 [2026-10-21 10:37:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6955 [2026-10-21 10:37:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4434 [2026-10-21 10:38:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2173 [2026-10-21 10:38:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1589 [2026-10-21 10:38:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3251 [2026-10-21 10:38:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1471 [2026-10-21 10:38:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4970 [2026-10-21 10:38:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0339 [2026-10-21 10:38:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0351 [2026-10-21 10:38:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6200 [2026-10-21 10:38:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6928 [2026-10-21 10:38:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2730 [2026-10-21 10:38:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9789 [2026-10-21 10:38:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1731 [2026-10-21 10:38:56][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,612 | MFU: 1.8% [2026-10-21 10:39:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0538 [2026-10-21 10:39:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4260 [2026-10-21 10:39:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4121 [2026-10-21 10:39:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4403 [2026-10-21 10:39:02][INFO][training.pretrain_engine] [Ph2|Step 96] loss=5.3205 | ppl=204.50 | aux=0.02508 | lr=2.99e-04 | gnorm=0.488 | tok/s=29,331 | MFU=7.0% | step=142998ms | tokens=0.903B | mem=4.2/32.7GB | cost=$7.88/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 00:09 (133.5h) | Phase3: SKIPPED | SFT: Oct 27 04:53 (138.2h) | DPO (Final): Oct 27 05:27 (138.8h) [2026-10-21 10:39:02][INFO][training.pretrain_engine] Starting global step 97 (accumulating 32 micro-batches)... [2026-10-21 10:39:02][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 10:39:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4652 [2026-10-21 10:39:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5384 [2026-10-21 10:39:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7947 [2026-10-21 10:39:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7033 [2026-10-21 10:39:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3766 [2026-10-21 10:39:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3903 [2026-10-21 10:39:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9068 [2026-10-21 10:39:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8806 [2026-10-21 10:39:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7809 [2026-10-21 10:39:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1875 [2026-10-21 10:39:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3939 [2026-10-21 10:39:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1200 [2026-10-21 10:40:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5222 [2026-10-21 10:40:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4044 [2026-10-21 10:40:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0678 [2026-10-21 10:40:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5700 [2026-10-21 10:40:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8227 [2026-10-21 10:40:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3843 [2026-10-21 10:40:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6692 [2026-10-21 10:40:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5659 [2026-10-21 10:40:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1992 [2026-10-21 10:40:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8885 [2026-10-21 10:40:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5818 [2026-10-21 10:40:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9075 [2026-10-21 10:41:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6231 [2026-10-21 10:41:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1819 [2026-10-21 10:41:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5154 [2026-10-21 10:41:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4114 [2026-10-21 10:41:19][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,627 | MFU: 1.8% [2026-10-21 10:41:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0820 [2026-10-21 10:41:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3272 [2026-10-21 10:41:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1709 [2026-10-21 10:41:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7074 [2026-10-21 10:41:25][INFO][training.pretrain_engine] [Ph2|Step 97] loss=5.4284 | ppl=227.78 | aux=0.02507 | lr=2.99e-04 | gnorm=0.447 | tok/s=29,330 | MFU=7.0% | step=143002ms | tokens=0.904B | mem=4.2/32.7GB | cost=$7.96/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 00:11 (133.5h) | Phase3: SKIPPED | SFT: Oct 27 04:55 (138.2h) | DPO (Final): Oct 27 05:29 (138.8h) [2026-10-21 10:41:25][INFO][training.pretrain_engine] Starting global step 98 (accumulating 32 micro-batches)... [2026-10-21 10:41:25][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 10:41:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1944 [2026-10-21 10:41:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4713 [2026-10-21 10:41:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2249 [2026-10-21 10:41:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1842 [2026-10-21 10:42:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3451 [2026-10-21 10:42:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4364 [2026-10-21 10:42:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2196 [2026-10-21 10:42:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4346 [2026-10-21 10:42:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4612 [2026-10-21 10:42:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8650 [2026-10-21 10:42:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6033 [2026-10-21 10:42:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8861 [2026-10-21 10:42:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3754 [2026-10-21 10:42:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2845 [2026-10-21 10:42:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9217 [2026-10-21 10:42:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2520 [2026-10-21 10:42:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8343 [2026-10-21 10:42:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1108 [2026-10-21 10:42:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3019 [2026-10-21 10:42:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1667 [2026-10-21 10:43:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7595 [2026-10-21 10:43:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6635 [2026-10-21 10:43:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5483 [2026-10-21 10:43:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9856 [2026-10-21 10:43:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2852 [2026-10-21 10:43:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2262 [2026-10-21 10:43:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6535 [2026-10-21 10:43:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5470 [2026-10-21 10:43:43][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,609 | MFU: 1.8% [2026-10-21 10:43:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7871 [2026-10-21 10:43:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6393 [2026-10-21 10:43:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5237 [2026-10-21 10:43:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1663 [2026-10-21 10:43:48][INFO][training.pretrain_engine] [Ph2|Step 98] loss=5.1902 | ppl=179.51 | aux=0.02508 | lr=2.99e-04 | gnorm=0.535 | tok/s=29,330 | MFU=7.0% | step=143003ms | tokens=0.905B | mem=4.2/32.7GB | cost=$8.04/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 00:13 (133.5h) | Phase3: SKIPPED | SFT: Oct 27 04:57 (138.2h) | DPO (Final): Oct 27 05:31 (138.8h) [2026-10-21 10:43:48][INFO][training.pretrain_engine] Starting global step 99 (accumulating 32 micro-batches)... [2026-10-21 10:43:48][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 10:44:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3611 [2026-10-21 10:44:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1770 [2026-10-21 10:44:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9565 [2026-10-21 10:44:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5296 [2026-10-21 10:44:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4065 [2026-10-21 10:44:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5703 [2026-10-21 10:44:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0034 [2026-10-21 10:44:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5620 [2026-10-21 10:44:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2943 [2026-10-21 10:44:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6157 [2026-10-21 10:44:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4130 [2026-10-21 10:44:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5783 [2026-10-21 10:44:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3390 [2026-10-21 10:44:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1694 [2026-10-21 10:45:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8528 [2026-10-21 10:45:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0393 [2026-10-21 10:45:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4136 [2026-10-21 10:45:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5144 [2026-10-21 10:45:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1933 [2026-10-21 10:45:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2940 [2026-10-21 10:45:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2740 [2026-10-21 10:45:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8748 [2026-10-21 10:45:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6513 [2026-10-21 10:45:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4781 [2026-10-21 10:45:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8691 [2026-10-21 10:45:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7283 [2026-10-21 10:45:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1103 [2026-10-21 10:45:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3544 [2026-10-21 10:46:06][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,608 | MFU: 1.8% [2026-10-21 10:46:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7415 [2026-10-21 10:46:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8217 [2026-10-21 10:46:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4968 [2026-10-21 10:46:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6433 [2026-10-21 10:46:11][INFO][training.pretrain_engine] [Ph2|Step 99] loss=5.2171 | ppl=184.39 | aux=0.02507 | lr=2.99e-04 | gnorm=0.471 | tok/s=29,330 | MFU=7.0% | step=143002ms | tokens=0.906B | mem=4.2/32.7GB | cost=$8.13/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 00:14 (133.5h) | Phase3: SKIPPED | SFT: Oct 27 04:59 (138.2h) | DPO (Final): Oct 27 05:33 (138.8h) [2026-10-21 10:46:14][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step99 | Tokens: 0.906B | Spent: $8.13 | Remaining: $991.87 [2026-10-21 10:46:14][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step99 [2026-10-21 10:46:14][INFO][training.pretrain_engine] Starting global step 100 (accumulating 32 micro-batches)... [2026-10-21 10:46:14][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 10:46:14][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 10:46:14][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 10:46:15][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 10:46:15][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step99 to Phase-Technologies/forge-500m-pretrain (path: phase2_step99)... [2026-10-21 10:46:15][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 10:46:25][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 10:46:25][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step99 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 10:46:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6455 [2026-10-21 10:46:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4352 [2026-10-21 10:46:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0577 [2026-10-21 10:46:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6208 [2026-10-21 10:46:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2184 [2026-10-21 10:46:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6587 [2026-10-21 10:46:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0680 [2026-10-21 10:46:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4026 [2026-10-21 10:47:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1390 [2026-10-21 10:47:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2137 [2026-10-21 10:47:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4633 [2026-10-21 10:47:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4218 [2026-10-21 10:47:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6794 [2026-10-21 10:47:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8922 [2026-10-21 10:47:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3957 [2026-10-21 10:47:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6957 [2026-10-21 10:47:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3673 [2026-10-21 10:47:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3358 [2026-10-21 10:47:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3139 [2026-10-21 10:47:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2110 [2026-10-21 10:47:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2053 [2026-10-21 10:47:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9755 [2026-10-21 10:47:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8327 [2026-10-21 10:48:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5989 [2026-10-21 10:48:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3192 [2026-10-21 10:48:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5242 [2026-10-21 10:48:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9150 [2026-10-21 10:48:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5419 [2026-10-21 10:48:32][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,633 | MFU: 1.8% [2026-10-21 10:48:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9706 [2026-10-21 10:48:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5584 [2026-10-21 10:48:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1109 [2026-10-21 10:48:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6039 [2026-10-21 10:48:36][INFO][training.pretrain_engine] [Ph2|Step 100] loss=5.2625 | ppl=192.97 | aux=0.02507 | lr=2.99e-04 | gnorm=0.494 | tok/s=29,333 | MFU=7.0% | step=142991ms | tokens=0.907B | mem=4.2/32.7GB | cost=$8.21/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 00:16 (133.5h) | Phase3: SKIPPED | SFT: Oct 27 05:00 (138.2h) | DPO (Final): Oct 27 05:34 (138.8h) [2026-10-21 10:48:36][INFO][training.pretrain_engine] Starting global step 101 (accumulating 32 micro-batches)... [2026-10-21 10:48:36][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 10:48:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8269 [2026-10-21 10:48:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5584 [2026-10-21 10:48:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1682 [2026-10-21 10:48:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2597 [2026-10-21 10:49:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5094 [2026-10-21 10:49:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5482 [2026-10-21 10:49:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2388 [2026-10-21 10:49:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1247 [2026-10-21 10:49:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6360 [2026-10-21 10:49:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6613 [2026-10-21 10:49:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5172 [2026-10-21 10:49:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1985 [2026-10-21 10:49:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2328 [2026-10-21 10:49:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5404 [2026-10-21 10:49:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3201 [2026-10-21 10:49:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3573 [2026-10-21 10:50:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2063 [2026-10-21 10:50:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8689 [2026-10-21 10:50:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1571 [2026-10-21 10:50:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5763 [2026-10-21 10:50:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7159 [2026-10-21 10:50:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6878 [2026-10-21 10:50:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3115 [2026-10-21 10:50:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2605 [2026-10-21 10:50:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6569 [2026-10-21 10:50:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9011 [2026-10-21 10:50:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4458 [2026-10-21 10:50:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6723 [2026-10-21 10:50:54][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,606 | MFU: 1.8% [2026-10-21 10:50:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8561 [2026-10-21 10:50:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7141 [2026-10-21 10:50:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8729 [2026-10-21 10:50:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8025 [2026-10-21 10:50:59][INFO][training.pretrain_engine] [Ph2|Step 101] loss=5.2440 | ppl=189.42 | aux=0.02508 | lr=2.99e-04 | gnorm=0.562 | tok/s=29,405 | MFU=7.1% | step=142640ms | tokens=0.908B | mem=4.2/32.7GB | cost=$8.29/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 26 23:58 (133.1h) | Phase3: SKIPPED | SFT: Oct 27 04:41 (137.8h) | DPO (Final): Oct 27 05:15 (138.4h) [2026-10-21 10:50:59][INFO][training.pretrain_engine] Starting global step 102 (accumulating 32 micro-batches)... [2026-10-21 10:50:59][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 10:51:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7696 [2026-10-21 10:51:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7127 [2026-10-21 10:51:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6026 [2026-10-21 10:51:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0255 [2026-10-21 10:51:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6420 [2026-10-21 10:51:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9271 [2026-10-21 10:51:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1306 [2026-10-21 10:51:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3071 [2026-10-21 10:51:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7280 [2026-10-21 10:51:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1258 [2026-10-21 10:51:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0936 [2026-10-21 10:51:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0455 [2026-10-21 10:52:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3671 [2026-10-21 10:52:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3671 [2026-10-21 10:52:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7636 [2026-10-21 10:52:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2475 [2026-10-21 10:52:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8239 [2026-10-21 10:52:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3793 [2026-10-21 10:52:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1781 [2026-10-21 10:52:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3570 [2026-10-21 10:52:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5832 [2026-10-21 10:52:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7196 [2026-10-21 10:52:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6513 [2026-10-21 10:52:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6625 [2026-10-21 10:53:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3702 [2026-10-21 10:53:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7168 [2026-10-21 10:53:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4439 [2026-10-21 10:53:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6601 [2026-10-21 10:53:17][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,614 | MFU: 1.8% [2026-10-21 10:53:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3832 [2026-10-21 10:53:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4220 [2026-10-21 10:53:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4770 [2026-10-21 10:53:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6015 [2026-10-21 10:53:22][INFO][training.pretrain_engine] [Ph2|Step 102] loss=5.2923 | ppl=198.79 | aux=0.02507 | lr=2.99e-04 | gnorm=0.488 | tok/s=29,403 | MFU=7.1% | step=142649ms | tokens=0.909B | mem=4.2/32.7GB | cost=$8.37/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 00:00 (133.1h) | Phase3: SKIPPED | SFT: Oct 27 04:44 (137.8h) | DPO (Final): Oct 27 05:18 (138.4h) [2026-10-21 10:53:22][INFO][training.pretrain_engine] Starting global step 103 (accumulating 32 micro-batches)... [2026-10-21 10:53:22][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 10:53:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7344 [2026-10-21 10:53:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3007 [2026-10-21 10:53:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8890 [2026-10-21 10:53:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8079 [2026-10-21 10:53:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6967 [2026-10-21 10:53:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3319 [2026-10-21 10:53:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2912 [2026-10-21 10:53:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3408 [2026-10-21 10:54:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1185 [2026-10-21 10:54:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4098 [2026-10-21 10:54:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2604 [2026-10-21 10:54:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1299 [2026-10-21 10:54:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7238 [2026-10-21 10:54:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7594 [2026-10-21 10:54:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6729 [2026-10-21 10:54:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9962 [2026-10-21 10:54:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1199 [2026-10-21 10:54:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3400 [2026-10-21 10:54:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2153 [2026-10-21 10:54:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4141 [2026-10-21 10:55:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5918 [2026-10-21 10:55:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7108 [2026-10-21 10:55:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5983 [2026-10-21 10:55:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3643 [2026-10-21 10:55:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5549 [2026-10-21 10:55:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4375 [2026-10-21 10:55:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2837 [2026-10-21 10:55:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3336 [2026-10-21 10:55:40][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,603 | MFU: 1.8% [2026-10-21 10:55:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8751 [2026-10-21 10:55:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4408 [2026-10-21 10:55:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4395 [2026-10-21 10:55:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5367 [2026-10-21 10:55:46][INFO][training.pretrain_engine] [Ph2|Step 103] loss=5.2346 | ppl=187.65 | aux=0.02507 | lr=2.99e-04 | gnorm=0.484 | tok/s=29,401 | MFU=7.1% | step=142659ms | tokens=0.910B | mem=4.2/32.7GB | cost=$8.45/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 00:02 (133.1h) | Phase3: SKIPPED | SFT: Oct 27 04:46 (137.8h) | DPO (Final): Oct 27 05:20 (138.4h) [2026-10-21 10:55:46][INFO][training.pretrain_engine] Starting global step 104 (accumulating 32 micro-batches)... [2026-10-21 10:55:46][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 10:56:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2308 [2026-10-21 10:56:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3692 [2026-10-21 10:56:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5162 [2026-10-21 10:56:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3483 [2026-10-21 10:56:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5149 [2026-10-21 10:56:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2360 [2026-10-21 10:56:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0562 [2026-10-21 10:56:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2465 [2026-10-21 10:56:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3042 [2026-10-21 10:56:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8340 [2026-10-21 10:56:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6789 [2026-10-21 10:56:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7609 [2026-10-21 10:56:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0938 [2026-10-21 10:56:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1975 [2026-10-21 10:56:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9476 [2026-10-21 10:56:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2297 [2026-10-21 10:57:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6237 [2026-10-21 10:57:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2683 [2026-10-21 10:57:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0115 [2026-10-21 10:57:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8978 [2026-10-21 10:57:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1746 [2026-10-21 10:57:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0679 [2026-10-21 10:57:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8703 [2026-10-21 10:57:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7354 [2026-10-21 10:57:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1343 [2026-10-21 10:57:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9229 [2026-10-21 10:57:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4860 [2026-10-21 10:57:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0254 [2026-10-21 10:58:03][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,631 | MFU: 1.8% [2026-10-21 10:58:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4295 [2026-10-21 10:58:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5548 [2026-10-21 10:58:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3705 [2026-10-21 10:58:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2530 [2026-10-21 10:58:10][INFO][training.pretrain_engine] [Ph2|Step 104] loss=5.3279 | ppl=206.01 | aux=0.02507 | lr=2.99e-04 | gnorm=0.512 | tok/s=29,398 | MFU=7.1% | step=142671ms | tokens=0.911B | mem=4.2/32.7GB | cost=$8.54/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 00:05 (133.1h) | Phase3: SKIPPED | SFT: Oct 27 04:48 (137.8h) | DPO (Final): Oct 27 05:22 (138.4h) [2026-10-21 10:58:10][INFO][training.pretrain_engine] Starting global step 105 (accumulating 32 micro-batches)... [2026-10-21 10:58:10][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 10:58:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4334 [2026-10-21 10:58:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2295 [2026-10-21 10:58:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8850 [2026-10-21 10:58:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9357 [2026-10-21 10:58:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5903 [2026-10-21 10:58:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8528 [2026-10-21 10:58:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9011 [2026-10-21 10:58:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1599 [2026-10-21 10:59:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9249 [2026-10-21 10:59:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7141 [2026-10-21 10:59:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4265 [2026-10-21 10:59:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7507 [2026-10-21 10:59:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8928 [2026-10-21 10:59:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0844 [2026-10-21 10:59:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0615 [2026-10-21 10:59:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3916 [2026-10-21 10:59:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2988 [2026-10-21 10:59:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2821 [2026-10-21 10:59:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9805 [2026-10-21 10:59:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4565 [2026-10-21 10:59:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9755 [2026-10-21 10:59:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4455 [2026-10-21 10:59:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5882 [2026-10-21 10:59:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9157 [2026-10-21 11:00:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7191 [2026-10-21 11:00:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6567 [2026-10-21 11:00:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1241 [2026-10-21 11:00:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7414 [2026-10-21 11:00:27][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,632 | MFU: 1.8% [2026-10-21 11:00:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4807 [2026-10-21 11:00:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3725 [2026-10-21 11:00:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3146 [2026-10-21 11:00:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2762 [2026-10-21 11:00:33][INFO][training.pretrain_engine] [Ph2|Step 105] loss=5.3478 | ppl=210.14 | aux=0.02506 | lr=2.99e-04 | gnorm=0.508 | tok/s=29,397 | MFU=7.1% | step=142678ms | tokens=0.912B | mem=4.2/32.7GB | cost=$8.62/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 00:07 (133.1h) | Phase3: SKIPPED | SFT: Oct 27 04:51 (137.8h) | DPO (Final): Oct 27 05:25 (138.4h) [2026-10-21 11:00:33][INFO][training.pretrain_engine] Starting global step 106 (accumulating 32 micro-batches)... [2026-10-21 11:00:33][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 11:00:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5381 [2026-10-21 11:00:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3795 [2026-10-21 11:00:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4356 [2026-10-21 11:00:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1282 [2026-10-21 11:01:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4696 [2026-10-21 11:01:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6144 [2026-10-21 11:01:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4364 [2026-10-21 11:01:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3531 [2026-10-21 11:01:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7024 [2026-10-21 11:01:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5824 [2026-10-21 11:01:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1594 [2026-10-21 11:01:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7769 [2026-10-21 11:01:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0559 [2026-10-21 11:01:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8335 [2026-10-21 11:01:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9541 [2026-10-21 11:01:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8652 [2026-10-21 11:02:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2702 [2026-10-21 11:02:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7058 [2026-10-21 11:02:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5105 [2026-10-21 11:02:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1574 [2026-10-21 11:02:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4361 [2026-10-21 11:02:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1550 [2026-10-21 11:02:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5414 [2026-10-21 11:02:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2820 [2026-10-21 11:02:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3334 [2026-10-21 11:02:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5424 [2026-10-21 11:02:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7189 [2026-10-21 11:02:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8549 [2026-10-21 11:02:50][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,630 | MFU: 1.8% [2026-10-21 11:02:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2660 [2026-10-21 11:02:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3831 [2026-10-21 11:02:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4551 [2026-10-21 11:02:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3499 [2026-10-21 11:02:56][INFO][training.pretrain_engine] [Ph2|Step 106] loss=5.3039 | ppl=201.13 | aux=0.02506 | lr=2.99e-04 | gnorm=0.609 | tok/s=29,396 | MFU=7.1% | step=142683ms | tokens=0.913B | mem=4.2/32.7GB | cost=$8.70/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 00:09 (133.1h) | Phase3: SKIPPED | SFT: Oct 27 04:53 (137.8h) | DPO (Final): Oct 27 05:27 (138.4h) [2026-10-21 11:02:56][INFO][training.pretrain_engine] Starting global step 107 (accumulating 32 micro-batches)... [2026-10-21 11:02:56][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 11:03:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4765 [2026-10-21 11:03:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3261 [2026-10-21 11:03:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8429 [2026-10-21 11:03:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6860 [2026-10-21 11:03:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3728 [2026-10-21 11:03:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2877 [2026-10-21 11:03:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4938 [2026-10-21 11:03:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9962 [2026-10-21 11:03:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9035 [2026-10-21 11:03:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7905 [2026-10-21 11:03:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8405 [2026-10-21 11:03:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2918 [2026-10-21 11:04:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8246 [2026-10-21 11:04:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4607 [2026-10-21 11:04:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1782 [2026-10-21 11:04:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3889 [2026-10-21 11:04:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0123 [2026-10-21 11:04:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3861 [2026-10-21 11:04:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7458 [2026-10-21 11:04:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6417 [2026-10-21 11:04:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4015 [2026-10-21 11:04:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2701 [2026-10-21 11:04:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8773 [2026-10-21 11:04:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5652 [2026-10-21 11:05:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0087 [2026-10-21 11:05:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5261 [2026-10-21 11:05:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2246 [2026-10-21 11:05:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4005 [2026-10-21 11:05:13][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,632 | MFU: 1.8% [2026-10-21 11:05:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3600 [2026-10-21 11:05:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9539 [2026-10-21 11:05:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8395 [2026-10-21 11:05:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5508 [2026-10-21 11:05:19][INFO][training.pretrain_engine] [Ph2|Step 107] loss=5.2792 | ppl=196.21 | aux=0.02508 | lr=2.99e-04 | gnorm=0.773 | tok/s=29,395 | MFU=7.1% | step=142688ms | tokens=0.914B | mem=4.2/32.7GB | cost=$8.78/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 00:11 (133.1h) | Phase3: SKIPPED | SFT: Oct 27 04:55 (137.8h) | DPO (Final): Oct 27 05:29 (138.4h) [2026-10-21 11:05:19][INFO][training.pretrain_engine] Starting global step 108 (accumulating 32 micro-batches)... [2026-10-21 11:05:19][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 11:05:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5819 [2026-10-21 11:05:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0841 [2026-10-21 11:05:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6030 [2026-10-21 11:05:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3195 [2026-10-21 11:05:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5275 [2026-10-21 11:05:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1023 [2026-10-21 11:05:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3331 [2026-10-21 11:05:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1670 [2026-10-21 11:06:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2222 [2026-10-21 11:06:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7491 [2026-10-21 11:06:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2700 [2026-10-21 11:06:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4269 [2026-10-21 11:06:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3625 [2026-10-21 11:06:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0429 [2026-10-21 11:06:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2467 [2026-10-21 11:06:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6517 [2026-10-21 11:06:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3837 [2026-10-21 11:06:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7190 [2026-10-21 11:06:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6435 [2026-10-21 11:06:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4401 [2026-10-21 11:07:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6965 [2026-10-21 11:07:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3375 [2026-10-21 11:07:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1854 [2026-10-21 11:07:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8379 [2026-10-21 11:07:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9921 [2026-10-21 11:07:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7597 [2026-10-21 11:07:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5755 [2026-10-21 11:07:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6887 [2026-10-21 11:07:36][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,632 | MFU: 1.8% [2026-10-21 11:07:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.1465 [2026-10-21 11:07:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.1517 [2026-10-21 11:07:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=3.7943 [2026-10-21 11:07:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4540 [2026-10-21 11:07:42][INFO][training.pretrain_engine] [Ph2|Step 108] loss=5.2344 | ppl=187.62 | aux=0.02508 | lr=2.99e-04 | gnorm=0.906 | tok/s=29,393 | MFU=7.1% | step=142696ms | tokens=0.915B | mem=4.2/32.7GB | cost=$8.86/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 00:13 (133.1h) | Phase3: SKIPPED | SFT: Oct 27 04:57 (137.8h) | DPO (Final): Oct 27 05:31 (138.4h) [2026-10-21 11:07:45][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step108 | Tokens: 0.915B | Spent: $8.87 | Remaining: $991.13 [2026-10-21 11:07:45][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step108 [2026-10-21 11:07:45][INFO][training.pretrain_engine] Starting global step 109 (accumulating 32 micro-batches)... [2026-10-21 11:07:45][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 11:07:45][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 11:07:45][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 11:07:46][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 11:07:46][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step108 to Phase-Technologies/forge-500m-pretrain (path: phase2_step108)... [2026-10-21 11:07:46][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 11:07:57][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 11:07:57][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step108 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 11:08:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0280 [2026-10-21 11:08:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1594 [2026-10-21 11:08:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9711 [2026-10-21 11:08:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1759 [2026-10-21 11:08:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1315 [2026-10-21 11:08:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4745 [2026-10-21 11:08:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4314 [2026-10-21 11:08:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9983 [2026-10-21 11:08:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4631 [2026-10-21 11:08:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3095 [2026-10-21 11:08:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7014 [2026-10-21 11:08:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4436 [2026-10-21 11:08:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2227 [2026-10-21 11:08:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7970 [2026-10-21 11:08:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5178 [2026-10-21 11:08:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2599 [2026-10-21 11:09:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3441 [2026-10-21 11:09:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1778 [2026-10-21 11:09:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9936 [2026-10-21 11:09:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3622 [2026-10-21 11:09:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3091 [2026-10-21 11:09:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3774 [2026-10-21 11:09:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2909 [2026-10-21 11:09:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5138 [2026-10-21 11:09:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0667 [2026-10-21 11:09:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0993 [2026-10-21 11:09:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2372 [2026-10-21 11:09:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3823 [2026-10-21 11:10:03][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,626 | MFU: 1.8% [2026-10-21 11:10:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5127 [2026-10-21 11:10:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1561 [2026-10-21 11:10:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3730 [2026-10-21 11:10:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6551 [2026-10-21 11:10:07][INFO][training.pretrain_engine] [Ph2|Step 109] loss=5.2827 | ppl=196.90 | aux=0.02507 | lr=2.99e-04 | gnorm=0.973 | tok/s=29,394 | MFU=7.1% | step=142693ms | tokens=0.916B | mem=4.2/32.7GB | cost=$8.95/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 00:15 (133.1h) | Phase3: SKIPPED | SFT: Oct 27 04:59 (137.8h) | DPO (Final): Oct 27 05:33 (138.4h) [2026-10-21 11:10:07][INFO][training.pretrain_engine] Starting global step 110 (accumulating 32 micro-batches)... [2026-10-21 11:10:07][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 11:10:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5540 [2026-10-21 11:10:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5584 [2026-10-21 11:10:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2376 [2026-10-21 11:10:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.3628 [2026-10-21 11:10:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8827 [2026-10-21 11:10:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9747 [2026-10-21 11:10:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3496 [2026-10-21 11:10:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8423 [2026-10-21 11:11:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9861 [2026-10-21 11:11:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2906 [2026-10-21 11:11:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4358 [2026-10-21 11:11:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2123 [2026-10-21 11:11:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1984 [2026-10-21 11:11:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2892 [2026-10-21 11:11:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2271 [2026-10-21 11:11:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5160 [2026-10-21 11:11:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5383 [2026-10-21 11:11:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2285 [2026-10-21 11:11:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0650 [2026-10-21 11:11:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2858 [2026-10-21 11:11:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1446 [2026-10-21 11:11:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5792 [2026-10-21 11:11:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7924 [2026-10-21 11:11:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6431 [2026-10-21 11:12:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7827 [2026-10-21 11:12:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3806 [2026-10-21 11:12:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6614 [2026-10-21 11:12:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8271 [2026-10-21 11:12:25][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,618 | MFU: 1.8% [2026-10-21 11:12:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1221 [2026-10-21 11:12:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8993 [2026-10-21 11:12:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2706 [2026-10-21 11:12:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0925 [2026-10-21 11:12:31][INFO][training.pretrain_engine] [Ph2|Step 110] loss=5.2808 | ppl=196.53 | aux=0.02508 | lr=2.99e-04 | gnorm=0.801 | tok/s=29,391 | MFU=7.1% | step=142706ms | tokens=0.918B | mem=4.2/32.7GB | cost=$9.03/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 00:18 (133.1h) | Phase3: SKIPPED | SFT: Oct 27 05:01 (137.8h) | DPO (Final): Oct 27 05:35 (138.4h) [2026-10-21 11:12:31][INFO][training.pretrain_engine] Starting global step 111 (accumulating 32 micro-batches)... [2026-10-21 11:12:31][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 11:12:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5596 [2026-10-21 11:12:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4975 [2026-10-21 11:12:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8662 [2026-10-21 11:12:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1415 [2026-10-21 11:13:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3813 [2026-10-21 11:13:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2407 [2026-10-21 11:13:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1888 [2026-10-21 11:13:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4878 [2026-10-21 11:13:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2062 [2026-10-21 11:13:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2368 [2026-10-21 11:13:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3217 [2026-10-21 11:13:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2533 [2026-10-21 11:13:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2836 [2026-10-21 11:13:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5016 [2026-10-21 11:13:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5849 [2026-10-21 11:13:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3682 [2026-10-21 11:13:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2251 [2026-10-21 11:13:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3959 [2026-10-21 11:14:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9876 [2026-10-21 11:14:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1657 [2026-10-21 11:14:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8270 [2026-10-21 11:14:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6360 [2026-10-21 11:14:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4748 [2026-10-21 11:14:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5806 [2026-10-21 11:14:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6224 [2026-10-21 11:14:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3155 [2026-10-21 11:14:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4361 [2026-10-21 11:14:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1678 [2026-10-21 11:14:48][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,620 | MFU: 1.8% [2026-10-21 11:14:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3141 [2026-10-21 11:14:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5503 [2026-10-21 11:14:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5580 [2026-10-21 11:14:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3088 [2026-10-21 11:14:54][INFO][training.pretrain_engine] [Ph2|Step 111] loss=5.4730 | ppl=238.17 | aux=0.02507 | lr=2.99e-04 | gnorm=0.742 | tok/s=29,390 | MFU=7.1% | step=142711ms | tokens=0.919B | mem=4.2/32.7GB | cost=$9.11/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 00:20 (133.1h) | Phase3: SKIPPED | SFT: Oct 27 05:03 (137.8h) | DPO (Final): Oct 27 05:37 (138.4h) [2026-10-21 11:14:54][INFO][training.pretrain_engine] Starting global step 112 (accumulating 32 micro-batches)... [2026-10-21 11:14:54][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 11:15:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5180 [2026-10-21 11:15:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8587 [2026-10-21 11:15:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2508 [2026-10-21 11:15:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8275 [2026-10-21 11:15:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8303 [2026-10-21 11:15:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5857 [2026-10-21 11:15:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8253 [2026-10-21 11:15:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4237 [2026-10-21 11:15:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4853 [2026-10-21 11:15:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4432 [2026-10-21 11:15:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0599 [2026-10-21 11:15:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6135 [2026-10-21 11:16:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8224 [2026-10-21 11:16:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1251 [2026-10-21 11:16:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8648 [2026-10-21 11:16:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9805 [2026-10-21 11:16:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0088 [2026-10-21 11:16:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3813 [2026-10-21 11:16:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5275 [2026-10-21 11:16:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5082 [2026-10-21 11:16:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4304 [2026-10-21 11:16:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2578 [2026-10-21 11:16:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2657 [2026-10-21 11:16:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0456 [2026-10-21 11:16:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7600 [2026-10-21 11:16:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5792 [2026-10-21 11:16:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2505 [2026-10-21 11:16:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5806 [2026-10-21 11:17:11][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,626 | MFU: 1.8% [2026-10-21 11:17:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4368 [2026-10-21 11:17:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6115 [2026-10-21 11:17:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5249 [2026-10-21 11:17:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3422 [2026-10-21 11:17:17][INFO][training.pretrain_engine] [Ph2|Step 112] loss=5.2008 | ppl=181.41 | aux=0.02507 | lr=2.99e-04 | gnorm=0.984 | tok/s=29,389 | MFU=7.1% | step=142718ms | tokens=0.920B | mem=4.2/32.7GB | cost=$9.19/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 00:22 (133.1h) | Phase3: SKIPPED | SFT: Oct 27 05:05 (137.8h) | DPO (Final): Oct 27 05:40 (138.4h) [2026-10-21 11:17:17][INFO][training.pretrain_engine] Starting global step 113 (accumulating 32 micro-batches)... [2026-10-21 11:17:17][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 11:17:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2205 [2026-10-21 11:17:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3029 [2026-10-21 11:17:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3534 [2026-10-21 11:17:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1620 [2026-10-21 11:17:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3519 [2026-10-21 11:17:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8461 [2026-10-21 11:17:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3805 [2026-10-21 11:17:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9429 [2026-10-21 11:18:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0387 [2026-10-21 11:18:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0300 [2026-10-21 11:18:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9146 [2026-10-21 11:18:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2617 [2026-10-21 11:18:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4652 [2026-10-21 11:18:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4407 [2026-10-21 11:18:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7378 [2026-10-21 11:18:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6274 [2026-10-21 11:18:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8852 [2026-10-21 11:18:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4591 [2026-10-21 11:18:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1706 [2026-10-21 11:18:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8980 [2026-10-21 11:19:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1534 [2026-10-21 11:19:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2467 [2026-10-21 11:19:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6360 [2026-10-21 11:19:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2081 [2026-10-21 11:19:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4648 [2026-10-21 11:19:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5060 [2026-10-21 11:19:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1038 [2026-10-21 11:19:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2612 [2026-10-21 11:19:34][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,622 | MFU: 1.8% [2026-10-21 11:19:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4366 [2026-10-21 11:19:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.1617 [2026-10-21 11:19:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1396 [2026-10-21 11:19:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1035 [2026-10-21 11:19:40][INFO][training.pretrain_engine] [Ph2|Step 113] loss=5.3359 | ppl=207.66 | aux=0.02508 | lr=2.99e-04 | gnorm=0.898 | tok/s=29,388 | MFU=7.1% | step=142722ms | tokens=0.921B | mem=4.2/32.7GB | cost=$9.28/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 00:24 (133.1h) | Phase3: SKIPPED | SFT: Oct 27 05:08 (137.8h) | DPO (Final): Oct 27 05:42 (138.4h) [2026-10-21 11:19:40][INFO][training.pretrain_engine] Starting global step 114 (accumulating 32 micro-batches)... [2026-10-21 11:19:40][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 11:19:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8623 [2026-10-21 11:19:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7501 [2026-10-21 11:19:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8704 [2026-10-21 11:19:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8498 [2026-10-21 11:20:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7972 [2026-10-21 11:20:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1573 [2026-10-21 11:20:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9188 [2026-10-21 11:20:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4191 [2026-10-21 11:20:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3050 [2026-10-21 11:20:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0637 [2026-10-21 11:20:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6707 [2026-10-21 11:20:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7034 [2026-10-21 11:20:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4532 [2026-10-21 11:20:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1079 [2026-10-21 11:20:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3513 [2026-10-21 11:20:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4238 [2026-10-21 11:21:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0300 [2026-10-21 11:21:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2796 [2026-10-21 11:21:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2447 [2026-10-21 11:21:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3648 [2026-10-21 11:21:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6496 [2026-10-21 11:21:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6115 [2026-10-21 11:21:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7640 [2026-10-21 11:21:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5703 [2026-10-21 11:21:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0675 [2026-10-21 11:21:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0713 [2026-10-21 11:21:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0745 [2026-10-21 11:21:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4192 [2026-10-21 11:21:58][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,614 | MFU: 1.8% [2026-10-21 11:22:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2471 [2026-10-21 11:22:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5464 [2026-10-21 11:22:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9174 [2026-10-21 11:22:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4829 [2026-10-21 11:22:03][INFO][training.pretrain_engine] [Ph2|Step 114] loss=5.3563 | ppl=211.95 | aux=0.02508 | lr=2.99e-04 | gnorm=0.957 | tok/s=29,388 | MFU=7.1% | step=142722ms | tokens=0.922B | mem=4.2/32.7GB | cost=$9.36/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 00:26 (133.1h) | Phase3: SKIPPED | SFT: Oct 27 05:09 (137.8h) | DPO (Final): Oct 27 05:43 (138.4h) [2026-10-21 11:22:03][INFO][training.pretrain_engine] Starting global step 115 (accumulating 32 micro-batches)... [2026-10-21 11:22:03][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 11:22:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2798 [2026-10-21 11:22:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2093 [2026-10-21 11:22:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4841 [2026-10-21 11:22:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0999 [2026-10-21 11:22:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4052 [2026-10-21 11:22:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4247 [2026-10-21 11:22:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4171 [2026-10-21 11:22:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1569 [2026-10-21 11:22:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0130 [2026-10-21 11:22:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.0932 [2026-10-21 11:22:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5669 [2026-10-21 11:22:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8942 [2026-10-21 11:23:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7687 [2026-10-21 11:23:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6246 [2026-10-21 11:23:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3762 [2026-10-21 11:23:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5811 [2026-10-21 11:23:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6351 [2026-10-21 11:23:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6923 [2026-10-21 11:23:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6440 [2026-10-21 11:23:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7136 [2026-10-21 11:23:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3767 [2026-10-21 11:23:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4292 [2026-10-21 11:23:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5392 [2026-10-21 11:23:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2458 [2026-10-21 11:24:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6333 [2026-10-21 11:24:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2978 [2026-10-21 11:24:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4043 [2026-10-21 11:24:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4456 [2026-10-21 11:24:20][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,609 | MFU: 1.8% [2026-10-21 11:24:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9693 [2026-10-21 11:24:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1167 [2026-10-21 11:24:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6203 [2026-10-21 11:24:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8077 [2026-10-21 11:24:25][INFO][training.pretrain_engine] [Ph2|Step 115] loss=5.4287 | ppl=227.86 | aux=0.02507 | lr=2.99e-04 | gnorm=0.691 | tok/s=29,389 | MFU=7.1% | step=142716ms | tokens=0.923B | mem=4.2/32.7GB | cost=$9.44/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 00:27 (133.1h) | Phase3: SKIPPED | SFT: Oct 27 05:11 (137.8h) | DPO (Final): Oct 27 05:45 (138.3h) [2026-10-21 11:24:25][INFO][training.pretrain_engine] Starting global step 116 (accumulating 32 micro-batches)... [2026-10-21 11:24:25][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 11:24:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5532 [2026-10-21 11:24:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9678 [2026-10-21 11:24:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4227 [2026-10-21 11:24:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2409 [2026-10-21 11:25:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0423 [2026-10-21 11:25:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1393 [2026-10-21 11:25:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4241 [2026-10-21 11:25:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3542 [2026-10-21 11:25:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8845 [2026-10-21 11:25:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3406 [2026-10-21 11:25:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3147 [2026-10-21 11:25:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9443 [2026-10-21 11:25:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3966 [2026-10-21 11:25:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3624 [2026-10-21 11:25:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5114 [2026-10-21 11:25:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4333 [2026-10-21 11:25:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4941 [2026-10-21 11:25:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9569 [2026-10-21 11:25:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9356 [2026-10-21 11:25:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0260 [2026-10-21 11:26:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7780 [2026-10-21 11:26:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3941 [2026-10-21 11:26:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2149 [2026-10-21 11:26:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5835 [2026-10-21 11:26:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7546 [2026-10-21 11:26:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9142 [2026-10-21 11:26:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0529 [2026-10-21 11:26:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4027 [2026-10-21 11:26:43][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,620 | MFU: 1.8% [2026-10-21 11:26:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5009 [2026-10-21 11:26:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5319 [2026-10-21 11:26:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3957 [2026-10-21 11:26:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1928 [2026-10-21 11:26:49][INFO][training.pretrain_engine] [Ph2|Step 116] loss=5.2683 | ppl=194.08 | aux=0.02507 | lr=2.99e-04 | gnorm=0.648 | tok/s=29,388 | MFU=7.1% | step=142720ms | tokens=0.924B | mem=4.2/32.7GB | cost=$9.52/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 00:29 (133.0h) | Phase3: SKIPPED | SFT: Oct 27 05:13 (137.8h) | DPO (Final): Oct 27 05:47 (138.3h) [2026-10-21 11:26:49][INFO][training.pretrain_engine] Starting global step 117 (accumulating 32 micro-batches)... [2026-10-21 11:26:49][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 11:27:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3232 [2026-10-21 11:27:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2678 [2026-10-21 11:27:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9013 [2026-10-21 11:27:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0974 [2026-10-21 11:27:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3771 [2026-10-21 11:27:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1458 [2026-10-21 11:27:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6645 [2026-10-21 11:27:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5533 [2026-10-21 11:27:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4095 [2026-10-21 11:27:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4510 [2026-10-21 11:27:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6332 [2026-10-21 11:27:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2187 [2026-10-21 11:27:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8420 [2026-10-21 11:27:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0077 [2026-10-21 11:28:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5737 [2026-10-21 11:28:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1151 [2026-10-21 11:28:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4622 [2026-10-21 11:28:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8272 [2026-10-21 11:28:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2071 [2026-10-21 11:28:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6174 [2026-10-21 11:28:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6138 [2026-10-21 11:28:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4138 [2026-10-21 11:28:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4935 [2026-10-21 11:28:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8797 [2026-10-21 11:28:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1070 [2026-10-21 11:28:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1851 [2026-10-21 11:28:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4836 [2026-10-21 11:28:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4254 [2026-10-21 11:29:06][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,628 | MFU: 1.8% [2026-10-21 11:29:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0987 [2026-10-21 11:29:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9191 [2026-10-21 11:29:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6415 [2026-10-21 11:29:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4964 [2026-10-21 11:29:12][INFO][training.pretrain_engine] [Ph2|Step 117] loss=5.3378 | ppl=208.06 | aux=0.02507 | lr=2.99e-04 | gnorm=0.605 | tok/s=29,388 | MFU=7.1% | step=142722ms | tokens=0.925B | mem=4.2/32.7GB | cost=$9.60/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 00:31 (133.0h) | Phase3: SKIPPED | SFT: Oct 27 05:15 (137.8h) | DPO (Final): Oct 27 05:49 (138.3h) [2026-10-21 11:29:15][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step117 | Tokens: 0.925B | Spent: $9.60 | Remaining: $990.40 [2026-10-21 11:29:15][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step117 [2026-10-21 11:29:15][INFO][training.pretrain_engine] Starting global step 118 (accumulating 32 micro-batches)... [2026-10-21 11:29:15][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 11:29:15][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 11:29:15][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 11:29:15][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 11:29:15][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step117 to Phase-Technologies/forge-500m-pretrain (path: phase2_step117)... [2026-10-21 11:29:15][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 11:29:26][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 11:29:26][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step117 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 11:29:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2212 [2026-10-21 11:29:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7079 [2026-10-21 11:29:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1389 [2026-10-21 11:29:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1800 [2026-10-21 11:29:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5339 [2026-10-21 11:29:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5876 [2026-10-21 11:29:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0982 [2026-10-21 11:29:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1271 [2026-10-21 11:30:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0625 [2026-10-21 11:30:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3606 [2026-10-21 11:30:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1503 [2026-10-21 11:30:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5078 [2026-10-21 11:30:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8606 [2026-10-21 11:30:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4979 [2026-10-21 11:30:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5204 [2026-10-21 11:30:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8237 [2026-10-21 11:30:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1371 [2026-10-21 11:30:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1254 [2026-10-21 11:30:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1573 [2026-10-21 11:30:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3412 [2026-10-21 11:30:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7874 [2026-10-21 11:30:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6285 [2026-10-21 11:30:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0329 [2026-10-21 11:31:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5916 [2026-10-21 11:31:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0789 [2026-10-21 11:31:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6328 [2026-10-21 11:31:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8210 [2026-10-21 11:31:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6214 [2026-10-21 11:31:33][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,607 | MFU: 1.8% [2026-10-21 11:31:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9412 [2026-10-21 11:31:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0542 [2026-10-21 11:31:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7385 [2026-10-21 11:31:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2725 [2026-10-21 11:31:37][INFO][training.pretrain_engine] [Ph2|Step 118] loss=5.2758 | ppl=195.55 | aux=0.02507 | lr=2.99e-04 | gnorm=0.629 | tok/s=29,389 | MFU=7.1% | step=142715ms | tokens=0.926B | mem=4.2/32.7GB | cost=$9.69/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 00:33 (133.0h) | Phase3: SKIPPED | SFT: Oct 27 05:16 (137.7h) | DPO (Final): Oct 27 05:50 (138.3h) [2026-10-21 11:31:37][INFO][training.pretrain_engine] Starting global step 119 (accumulating 32 micro-batches)... [2026-10-21 11:31:37][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 11:31:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1092 [2026-10-21 11:31:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0607 [2026-10-21 11:31:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4366 [2026-10-21 11:31:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6200 [2026-10-21 11:32:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0925 [2026-10-21 11:32:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4949 [2026-10-21 11:32:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9972 [2026-10-21 11:32:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9515 [2026-10-21 11:32:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4820 [2026-10-21 11:32:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5973 [2026-10-21 11:32:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4023 [2026-10-21 11:32:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3848 [2026-10-21 11:32:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0845 [2026-10-21 11:32:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6639 [2026-10-21 11:32:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8908 [2026-10-21 11:32:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5340 [2026-10-21 11:33:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0807 [2026-10-21 11:33:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1635 [2026-10-21 11:33:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3004 [2026-10-21 11:33:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1448 [2026-10-21 11:33:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7807 [2026-10-21 11:33:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7658 [2026-10-21 11:33:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8731 [2026-10-21 11:33:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7816 [2026-10-21 11:33:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6358 [2026-10-21 11:33:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2543 [2026-10-21 11:33:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8110 [2026-10-21 11:33:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9820 [2026-10-21 11:33:55][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,601 | MFU: 1.8% [2026-10-21 11:34:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6992 [2026-10-21 11:34:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7945 [2026-10-21 11:34:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1870 [2026-10-21 11:34:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4322 [2026-10-21 11:34:00][INFO][training.pretrain_engine] [Ph2|Step 119] loss=5.3472 | ppl=210.02 | aux=0.02506 | lr=2.99e-04 | gnorm=0.523 | tok/s=29,388 | MFU=7.1% | step=142723ms | tokens=0.927B | mem=4.2/32.7GB | cost=$9.77/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 00:35 (133.0h) | Phase3: SKIPPED | SFT: Oct 27 05:18 (137.7h) | DPO (Final): Oct 27 05:52 (138.3h) [2026-10-21 11:34:00][INFO][training.pretrain_engine] Starting global step 120 (accumulating 32 micro-batches)... [2026-10-21 11:34:00][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 11:34:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3729 [2026-10-21 11:34:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4366 [2026-10-21 11:34:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4872 [2026-10-21 11:34:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3929 [2026-10-21 11:34:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8386 [2026-10-21 11:34:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9314 [2026-10-21 11:34:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8983 [2026-10-21 11:34:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.2363 [2026-10-21 11:34:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2930 [2026-10-21 11:34:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0095 [2026-10-21 11:34:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4588 [2026-10-21 11:34:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1731 [2026-10-21 11:35:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3827 [2026-10-21 11:35:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3429 [2026-10-21 11:35:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3605 [2026-10-21 11:35:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5128 [2026-10-21 11:35:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0758 [2026-10-21 11:35:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3195 [2026-10-21 11:35:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2885 [2026-10-21 11:35:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6642 [2026-10-21 11:35:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3561 [2026-10-21 11:35:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3946 [2026-10-21 11:35:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7431 [2026-10-21 11:35:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6422 [2026-10-21 11:36:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2487 [2026-10-21 11:36:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9404 [2026-10-21 11:36:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8093 [2026-10-21 11:36:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.1873 [2026-10-21 11:36:18][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,614 | MFU: 1.8% [2026-10-21 11:36:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0138 [2026-10-21 11:36:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3687 [2026-10-21 11:36:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6331 [2026-10-21 11:36:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6651 [2026-10-21 11:36:22][INFO][training.pretrain_engine] [Ph2|Step 120] loss=5.3439 | ppl=209.32 | aux=0.02506 | lr=2.99e-04 | gnorm=0.504 | tok/s=29,388 | MFU=7.1% | step=142721ms | tokens=0.928B | mem=4.2/32.7GB | cost=$9.85/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 00:36 (133.0h) | Phase3: SKIPPED | SFT: Oct 27 05:20 (137.7h) | DPO (Final): Oct 27 05:54 (138.3h) [2026-10-21 11:36:22][INFO][training.pretrain_engine] Starting global step 121 (accumulating 32 micro-batches)... [2026-10-21 11:36:22][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 11:36:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0501 [2026-10-21 11:36:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5241 [2026-10-21 11:36:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3444 [2026-10-21 11:36:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2601 [2026-10-21 11:36:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2084 [2026-10-21 11:36:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8793 [2026-10-21 11:36:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1472 [2026-10-21 11:36:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1164 [2026-10-21 11:37:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2199 [2026-10-21 11:37:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4702 [2026-10-21 11:37:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2891 [2026-10-21 11:37:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8396 [2026-10-21 11:37:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4075 [2026-10-21 11:37:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4083 [2026-10-21 11:37:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2434 [2026-10-21 11:37:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4349 [2026-10-21 11:37:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7925 [2026-10-21 11:37:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0171 [2026-10-21 11:37:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8771 [2026-10-21 11:37:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9906 [2026-10-21 11:38:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5751 [2026-10-21 11:38:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4562 [2026-10-21 11:38:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5922 [2026-10-21 11:38:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3868 [2026-10-21 11:38:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3788 [2026-10-21 11:38:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2561 [2026-10-21 11:38:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0840 [2026-10-21 11:38:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5017 [2026-10-21 11:38:40][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,610 | MFU: 1.8% [2026-10-21 11:38:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3338 [2026-10-21 11:38:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3893 [2026-10-21 11:38:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9908 [2026-10-21 11:38:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5079 [2026-10-21 11:38:45][INFO][training.pretrain_engine] [Ph2|Step 121] loss=5.2373 | ppl=188.16 | aux=0.02506 | lr=2.99e-04 | gnorm=0.531 | tok/s=29,388 | MFU=7.1% | step=142721ms | tokens=0.929B | mem=4.2/32.7GB | cost=$9.93/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 00:38 (133.0h) | Phase3: SKIPPED | SFT: Oct 27 05:22 (137.7h) | DPO (Final): Oct 27 05:56 (138.3h) [2026-10-21 11:38:45][INFO][training.pretrain_engine] Starting global step 122 (accumulating 32 micro-batches)... [2026-10-21 11:38:45][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 11:39:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7952 [2026-10-21 11:39:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3377 [2026-10-21 11:39:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8791 [2026-10-21 11:39:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2632 [2026-10-21 11:39:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7180 [2026-10-21 11:39:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9428 [2026-10-21 11:39:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8673 [2026-10-21 11:39:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5888 [2026-10-21 11:39:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5375 [2026-10-21 11:39:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3474 [2026-10-21 11:39:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1881 [2026-10-21 11:39:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2930 [2026-10-21 11:39:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2529 [2026-10-21 11:39:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4904 [2026-10-21 11:39:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1985 [2026-10-21 11:39:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0101 [2026-10-21 11:40:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4781 [2026-10-21 11:40:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3103 [2026-10-21 11:40:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7739 [2026-10-21 11:40:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3468 [2026-10-21 11:40:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6463 [2026-10-21 11:40:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3624 [2026-10-21 11:40:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4742 [2026-10-21 11:40:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6540 [2026-10-21 11:40:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7790 [2026-10-21 11:40:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1490 [2026-10-21 11:40:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6979 [2026-10-21 11:40:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2799 [2026-10-21 11:41:03][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,614 | MFU: 1.8% [2026-10-21 11:41:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0989 [2026-10-21 11:41:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4063 [2026-10-21 11:41:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2794 [2026-10-21 11:41:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4126 [2026-10-21 11:41:08][INFO][training.pretrain_engine] [Ph2|Step 122] loss=5.2721 | ppl=194.83 | aux=0.02506 | lr=2.99e-04 | gnorm=0.498 | tok/s=29,389 | MFU=7.1% | step=142716ms | tokens=0.930B | mem=4.2/32.7GB | cost=$10.01/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 00:40 (133.0h) | Phase3: SKIPPED | SFT: Oct 27 05:23 (137.7h) | DPO (Final): Oct 27 05:57 (138.3h) [2026-10-21 11:41:08][INFO][training.pretrain_engine] Starting global step 123 (accumulating 32 micro-batches)... [2026-10-21 11:41:08][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 11:41:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2708 [2026-10-21 11:41:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8678 [2026-10-21 11:41:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1415 [2026-10-21 11:41:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6661 [2026-10-21 11:41:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6758 [2026-10-21 11:41:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5969 [2026-10-21 11:41:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5227 [2026-10-21 11:41:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4742 [2026-10-21 11:42:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5527 [2026-10-21 11:42:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5899 [2026-10-21 11:42:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4701 [2026-10-21 11:42:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4846 [2026-10-21 11:42:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0447 [2026-10-21 11:42:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3021 [2026-10-21 11:42:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1258 [2026-10-21 11:42:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3100 [2026-10-21 11:42:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6984 [2026-10-21 11:42:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7746 [2026-10-21 11:42:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6185 [2026-10-21 11:42:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7514 [2026-10-21 11:42:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2734 [2026-10-21 11:42:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2967 [2026-10-21 11:42:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6974 [2026-10-21 11:42:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2379 [2026-10-21 11:43:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2351 [2026-10-21 11:43:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1602 [2026-10-21 11:43:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7309 [2026-10-21 11:43:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2883 [2026-10-21 11:43:25][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,628 | MFU: 1.8% [2026-10-21 11:43:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0701 [2026-10-21 11:43:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6055 [2026-10-21 11:43:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3118 [2026-10-21 11:43:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2731 [2026-10-21 11:43:31][INFO][training.pretrain_engine] [Ph2|Step 123] loss=5.4108 | ppl=223.81 | aux=0.02507 | lr=2.99e-04 | gnorm=0.523 | tok/s=29,390 | MFU=7.1% | step=142714ms | tokens=0.931B | mem=4.2/32.7GB | cost=$10.09/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 00:41 (133.0h) | Phase3: SKIPPED | SFT: Oct 27 05:25 (137.7h) | DPO (Final): Oct 27 05:59 (138.3h) [2026-10-21 11:43:31][INFO][training.pretrain_engine] Starting global step 124 (accumulating 32 micro-batches)... [2026-10-21 11:43:31][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 11:43:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7005 [2026-10-21 11:43:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2560 [2026-10-21 11:43:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1285 [2026-10-21 11:43:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7003 [2026-10-21 11:44:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2706 [2026-10-21 11:44:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1912 [2026-10-21 11:44:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4473 [2026-10-21 11:44:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3889 [2026-10-21 11:44:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9910 [2026-10-21 11:44:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4110 [2026-10-21 11:44:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8055 [2026-10-21 11:44:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8025 [2026-10-21 11:44:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3699 [2026-10-21 11:44:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2784 [2026-10-21 11:44:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5425 [2026-10-21 11:44:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2606 [2026-10-21 11:44:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=3.7447 [2026-10-21 11:45:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9645 [2026-10-21 11:45:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0878 [2026-10-21 11:45:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.1404 [2026-10-21 11:45:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9794 [2026-10-21 11:45:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8881 [2026-10-21 11:45:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6766 [2026-10-21 11:45:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7360 [2026-10-21 11:45:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2482 [2026-10-21 11:45:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4621 [2026-10-21 11:45:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4092 [2026-10-21 11:45:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7250 [2026-10-21 11:45:48][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,614 | MFU: 1.8% [2026-10-21 11:45:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1858 [2026-10-21 11:45:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1118 [2026-10-21 11:45:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9636 [2026-10-21 11:45:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1476 [2026-10-21 11:45:53][INFO][training.pretrain_engine] [Ph2|Step 124] loss=5.1876 | ppl=179.04 | aux=0.02508 | lr=2.99e-04 | gnorm=0.551 | tok/s=29,390 | MFU=7.1% | step=142711ms | tokens=0.932B | mem=4.2/32.7GB | cost=$10.18/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 00:43 (133.0h) | Phase3: SKIPPED | SFT: Oct 27 05:27 (137.7h) | DPO (Final): Oct 27 06:01 (138.3h) [2026-10-21 11:45:53][INFO][training.pretrain_engine] Starting global step 125 (accumulating 32 micro-batches)... [2026-10-21 11:45:53][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 11:46:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1376 [2026-10-21 11:46:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5582 [2026-10-21 11:46:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6640 [2026-10-21 11:46:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2051 [2026-10-21 11:46:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2541 [2026-10-21 11:46:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3516 [2026-10-21 11:46:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3681 [2026-10-21 11:46:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5231 [2026-10-21 11:46:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8451 [2026-10-21 11:46:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5713 [2026-10-21 11:46:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0356 [2026-10-21 11:46:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6706 [2026-10-21 11:47:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5358 [2026-10-21 11:47:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6242 [2026-10-21 11:47:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0358 [2026-10-21 11:47:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7870 [2026-10-21 11:47:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6604 [2026-10-21 11:47:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2714 [2026-10-21 11:47:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3791 [2026-10-21 11:47:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7515 [2026-10-21 11:47:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0164 [2026-10-21 11:47:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2938 [2026-10-21 11:47:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2050 [2026-10-21 11:47:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3263 [2026-10-21 11:47:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0134 [2026-10-21 11:47:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0494 [2026-10-21 11:47:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1520 [2026-10-21 11:47:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6598 [2026-10-21 11:48:11][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,618 | MFU: 1.8% [2026-10-21 11:48:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8001 [2026-10-21 11:48:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4449 [2026-10-21 11:48:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7382 [2026-10-21 11:48:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0337 [2026-10-21 11:48:16][INFO][training.pretrain_engine] [Ph2|Step 125] loss=5.3212 | ppl=204.64 | aux=0.02506 | lr=2.99e-04 | gnorm=0.516 | tok/s=29,391 | MFU=7.1% | step=142709ms | tokens=0.933B | mem=4.2/32.7GB | cost=$10.26/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 00:45 (132.9h) | Phase3: SKIPPED | SFT: Oct 27 05:28 (137.7h) | DPO (Final): Oct 27 06:02 (138.2h) [2026-10-21 11:48:16][INFO][training.pretrain_engine] Starting global step 126 (accumulating 32 micro-batches)... [2026-10-21 11:48:16][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 11:48:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7469 [2026-10-21 11:48:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7216 [2026-10-21 11:48:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1303 [2026-10-21 11:48:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8313 [2026-10-21 11:48:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4290 [2026-10-21 11:48:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4366 [2026-10-21 11:48:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6152 [2026-10-21 11:48:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0321 [2026-10-21 11:49:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4694 [2026-10-21 11:49:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2844 [2026-10-21 11:49:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3523 [2026-10-21 11:49:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3666 [2026-10-21 11:49:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6440 [2026-10-21 11:49:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1606 [2026-10-21 11:49:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5779 [2026-10-21 11:49:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3164 [2026-10-21 11:49:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2279 [2026-10-21 11:49:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2944 [2026-10-21 11:49:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1178 [2026-10-21 11:49:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5260 [2026-10-21 11:50:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1704 [2026-10-21 11:50:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2710 [2026-10-21 11:50:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1449 [2026-10-21 11:50:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0010 [2026-10-21 11:50:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2793 [2026-10-21 11:50:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2685 [2026-10-21 11:50:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4715 [2026-10-21 11:50:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2116 [2026-10-21 11:50:34][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,603 | MFU: 1.8% [2026-10-21 11:50:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3779 [2026-10-21 11:50:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5046 [2026-10-21 11:50:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4628 [2026-10-21 11:50:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5334 [2026-10-21 11:50:39][INFO][training.pretrain_engine] [Ph2|Step 126] loss=5.3574 | ppl=212.16 | aux=0.02507 | lr=2.99e-04 | gnorm=0.508 | tok/s=29,392 | MFU=7.1% | step=142701ms | tokens=0.934B | mem=4.2/32.7GB | cost=$10.34/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 00:46 (132.9h) | Phase3: SKIPPED | SFT: Oct 27 05:30 (137.7h) | DPO (Final): Oct 27 06:04 (138.2h) [2026-10-21 11:50:42][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step126 | Tokens: 0.934B | Spent: $10.34 | Remaining: $989.66 [2026-10-21 11:50:42][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step126 [2026-10-21 11:50:42][INFO][training.pretrain_engine] Starting global step 127 (accumulating 32 micro-batches)... [2026-10-21 11:50:42][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 11:50:42][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 11:50:42][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 11:50:42][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 11:50:42][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step126 to Phase-Technologies/forge-500m-pretrain (path: phase2_step126)... [2026-10-21 11:50:42][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 11:50:56][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 11:50:56][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step126 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 11:50:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3146 [2026-10-21 11:50:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4758 [2026-10-21 11:50:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2917 [2026-10-21 11:51:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6279 [2026-10-21 11:51:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2572 [2026-10-21 11:51:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2785 [2026-10-21 11:51:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2862 [2026-10-21 11:51:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7611 [2026-10-21 11:51:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6095 [2026-10-21 11:51:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6320 [2026-10-21 11:51:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5999 [2026-10-21 11:51:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2297 [2026-10-21 11:51:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0844 [2026-10-21 11:51:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6983 [2026-10-21 11:51:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2200 [2026-10-21 11:51:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2002 [2026-10-21 11:52:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4924 [2026-10-21 11:52:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1645 [2026-10-21 11:52:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4662 [2026-10-21 11:52:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5539 [2026-10-21 11:52:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7445 [2026-10-21 11:52:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3239 [2026-10-21 11:52:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2679 [2026-10-21 11:52:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6120 [2026-10-21 11:52:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.1407 [2026-10-21 11:52:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0224 [2026-10-21 11:52:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4349 [2026-10-21 11:52:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0741 [2026-10-21 11:53:00][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,609 | MFU: 1.8% [2026-10-21 11:53:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1853 [2026-10-21 11:53:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9780 [2026-10-21 11:53:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9950 [2026-10-21 11:53:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8531 [2026-10-21 11:53:04][INFO][training.pretrain_engine] [Ph2|Step 127] loss=5.2798 | ppl=196.33 | aux=0.02508 | lr=2.99e-04 | gnorm=0.570 | tok/s=29,395 | MFU=7.1% | step=142689ms | tokens=0.935B | mem=4.2/32.7GB | cost=$10.42/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 00:47 (132.9h) | Phase3: SKIPPED | SFT: Oct 27 05:31 (137.6h) | DPO (Final): Oct 27 06:05 (138.2h) [2026-10-21 11:53:04][INFO][training.pretrain_engine] Starting global step 128 (accumulating 32 micro-batches)... [2026-10-21 11:53:04][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 11:53:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9596 [2026-10-21 11:53:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2779 [2026-10-21 11:53:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1083 [2026-10-21 11:53:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0455 [2026-10-21 11:53:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2466 [2026-10-21 11:53:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3439 [2026-10-21 11:53:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4240 [2026-10-21 11:53:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8354 [2026-10-21 11:53:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4153 [2026-10-21 11:53:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1677 [2026-10-21 11:53:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4387 [2026-10-21 11:53:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2387 [2026-10-21 11:54:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5268 [2026-10-21 11:54:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6120 [2026-10-21 11:54:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5107 [2026-10-21 11:54:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2652 [2026-10-21 11:54:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5064 [2026-10-21 11:54:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1684 [2026-10-21 11:54:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2761 [2026-10-21 11:54:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5067 [2026-10-21 11:54:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2880 [2026-10-21 11:54:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7478 [2026-10-21 11:54:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6849 [2026-10-21 11:54:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4089 [2026-10-21 11:55:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3178 [2026-10-21 11:55:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2284 [2026-10-21 11:55:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2382 [2026-10-21 11:55:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8999 [2026-10-21 11:55:22][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,629 | MFU: 1.8% [2026-10-21 11:55:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9764 [2026-10-21 11:55:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9167 [2026-10-21 11:55:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3579 [2026-10-21 11:55:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3836 [2026-10-21 11:55:27][INFO][training.pretrain_engine] [Ph2|Step 128] loss=5.2720 | ppl=194.81 | aux=0.02507 | lr=2.99e-04 | gnorm=0.570 | tok/s=29,394 | MFU=7.1% | step=142693ms | tokens=0.936B | mem=4.2/32.7GB | cost=$10.50/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 00:49 (132.9h) | Phase3: SKIPPED | SFT: Oct 27 05:33 (137.6h) | DPO (Final): Oct 27 06:07 (138.2h) [2026-10-21 11:55:27][INFO][training.pretrain_engine] Starting global step 129 (accumulating 32 micro-batches)... [2026-10-21 11:55:27][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 11:55:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2027 [2026-10-21 11:55:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9245 [2026-10-21 11:55:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9710 [2026-10-21 11:55:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8740 [2026-10-21 11:56:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0378 [2026-10-21 11:56:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3552 [2026-10-21 11:56:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5062 [2026-10-21 11:56:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9854 [2026-10-21 11:56:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3635 [2026-10-21 11:56:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8750 [2026-10-21 11:56:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5053 [2026-10-21 11:56:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1872 [2026-10-21 11:56:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9113 [2026-10-21 11:56:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6301 [2026-10-21 11:56:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8312 [2026-10-21 11:56:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2302 [2026-10-21 11:56:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4367 [2026-10-21 11:56:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5481 [2026-10-21 11:56:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2327 [2026-10-21 11:56:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4305 [2026-10-21 11:57:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5580 [2026-10-21 11:57:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1315 [2026-10-21 11:57:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2735 [2026-10-21 11:57:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5126 [2026-10-21 11:57:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2913 [2026-10-21 11:57:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2970 [2026-10-21 11:57:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9828 [2026-10-21 11:57:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3576 [2026-10-21 11:57:44][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,621 | MFU: 1.8% [2026-10-21 11:57:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4044 [2026-10-21 11:57:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2581 [2026-10-21 11:57:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3013 [2026-10-21 11:57:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0795 [2026-10-21 11:57:49][INFO][training.pretrain_engine] [Ph2|Step 129] loss=5.3866 | ppl=218.45 | aux=0.02507 | lr=2.99e-04 | gnorm=0.633 | tok/s=29,393 | MFU=7.1% | step=142695ms | tokens=0.937B | mem=4.2/32.7GB | cost=$10.59/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 00:51 (132.9h) | Phase3: SKIPPED | SFT: Oct 27 05:35 (137.6h) | DPO (Final): Oct 27 06:09 (138.2h) [2026-10-21 11:57:49][INFO][training.pretrain_engine] Starting global step 130 (accumulating 32 micro-batches)... [2026-10-21 11:57:49][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 11:58:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1179 [2026-10-21 11:58:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3387 [2026-10-21 11:58:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0657 [2026-10-21 11:58:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3754 [2026-10-21 11:58:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2110 [2026-10-21 11:58:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2630 [2026-10-21 11:58:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1067 [2026-10-21 11:58:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1951 [2026-10-21 11:58:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6710 [2026-10-21 11:58:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9672 [2026-10-21 11:58:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1246 [2026-10-21 11:58:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1689 [2026-10-21 11:59:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0877 [2026-10-21 11:59:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3320 [2026-10-21 11:59:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4519 [2026-10-21 11:59:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2898 [2026-10-21 11:59:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8858 [2026-10-21 11:59:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0683 [2026-10-21 11:59:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3435 [2026-10-21 11:59:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1852 [2026-10-21 11:59:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2551 [2026-10-21 11:59:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0307 [2026-10-21 11:59:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3224 [2026-10-21 11:59:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4218 [2026-10-21 11:59:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2274 [2026-10-21 11:59:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3736 [2026-10-21 11:59:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3848 [2026-10-21 11:59:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2015 [2026-10-21 12:00:07][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,619 | MFU: 1.8% [2026-10-21 12:00:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6122 [2026-10-21 12:00:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7232 [2026-10-21 12:00:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9120 [2026-10-21 12:00:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3205 [2026-10-21 12:00:12][INFO][training.pretrain_engine] [Ph2|Step 130] loss=5.1540 | ppl=173.12 | aux=0.02506 | lr=2.99e-04 | gnorm=0.727 | tok/s=29,392 | MFU=7.1% | step=142702ms | tokens=0.938B | mem=4.2/32.7GB | cost=$10.67/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 00:53 (132.9h) | Phase3: SKIPPED | SFT: Oct 27 05:37 (137.6h) | DPO (Final): Oct 27 06:11 (138.2h) [2026-10-21 12:00:12][INFO][training.pretrain_engine] Starting global step 131 (accumulating 32 micro-batches)... [2026-10-21 12:00:12][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 12:00:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5564 [2026-10-21 12:00:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2610 [2026-10-21 12:00:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4334 [2026-10-21 12:00:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8720 [2026-10-21 12:00:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4823 [2026-10-21 12:00:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4560 [2026-10-21 12:00:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1330 [2026-10-21 12:00:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3724 [2026-10-21 12:01:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0767 [2026-10-21 12:01:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3714 [2026-10-21 12:01:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7944 [2026-10-21 12:01:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4358 [2026-10-21 12:01:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3292 [2026-10-21 12:01:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3672 [2026-10-21 12:01:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3499 [2026-10-21 12:01:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2563 [2026-10-21 12:01:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1859 [2026-10-21 12:01:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2892 [2026-10-21 12:01:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2970 [2026-10-21 12:01:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4539 [2026-10-21 12:01:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0331 [2026-10-21 12:01:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9411 [2026-10-21 12:01:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8905 [2026-10-21 12:02:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1268 [2026-10-21 12:02:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8381 [2026-10-21 12:02:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0595 [2026-10-21 12:02:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4766 [2026-10-21 12:02:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0162 [2026-10-21 12:02:30][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,629 | MFU: 1.8% [2026-10-21 12:02:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4534 [2026-10-21 12:02:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3608 [2026-10-21 12:02:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5128 [2026-10-21 12:02:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2964 [2026-10-21 12:02:35][INFO][training.pretrain_engine] [Ph2|Step 131] loss=5.2579 | ppl=192.08 | aux=0.02508 | lr=2.99e-04 | gnorm=0.855 | tok/s=29,391 | MFU=7.1% | step=142707ms | tokens=0.940B | mem=4.2/32.7GB | cost=$10.75/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 00:55 (132.9h) | Phase3: SKIPPED | SFT: Oct 27 05:39 (137.6h) | DPO (Final): Oct 27 06:13 (138.2h) [2026-10-21 12:02:35][INFO][training.pretrain_engine] Starting global step 132 (accumulating 32 micro-batches)... [2026-10-21 12:02:35][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 12:02:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3967 [2026-10-21 12:02:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9486 [2026-10-21 12:02:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2272 [2026-10-21 12:02:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3367 [2026-10-21 12:03:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3167 [2026-10-21 12:03:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8635 [2026-10-21 12:03:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2497 [2026-10-21 12:03:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5212 [2026-10-21 12:03:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8965 [2026-10-21 12:03:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3754 [2026-10-21 12:03:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9140 [2026-10-21 12:03:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5151 [2026-10-21 12:03:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4245 [2026-10-21 12:03:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4320 [2026-10-21 12:03:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4208 [2026-10-21 12:03:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7840 [2026-10-21 12:04:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2412 [2026-10-21 12:04:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3160 [2026-10-21 12:04:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0893 [2026-10-21 12:04:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2370 [2026-10-21 12:04:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5877 [2026-10-21 12:04:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5328 [2026-10-21 12:04:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3728 [2026-10-21 12:04:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3180 [2026-10-21 12:04:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7725 [2026-10-21 12:04:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6424 [2026-10-21 12:04:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2888 [2026-10-21 12:04:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8422 [2026-10-21 12:04:53][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,607 | MFU: 1.8% [2026-10-21 12:04:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2521 [2026-10-21 12:04:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4137 [2026-10-21 12:04:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4382 [2026-10-21 12:04:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1446 [2026-10-21 12:04:58][INFO][training.pretrain_engine] [Ph2|Step 132] loss=5.3369 | ppl=207.86 | aux=0.02507 | lr=2.99e-04 | gnorm=0.867 | tok/s=29,390 | MFU=7.1% | step=142710ms | tokens=0.941B | mem=4.2/32.7GB | cost=$10.83/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 00:57 (132.9h) | Phase3: SKIPPED | SFT: Oct 27 05:41 (137.6h) | DPO (Final): Oct 27 06:15 (138.2h) [2026-10-21 12:04:58][INFO][training.pretrain_engine] Starting global step 133 (accumulating 32 micro-batches)... [2026-10-21 12:04:58][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 12:05:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2724 [2026-10-21 12:05:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4876 [2026-10-21 12:05:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9678 [2026-10-21 12:05:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8200 [2026-10-21 12:05:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2992 [2026-10-21 12:05:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4107 [2026-10-21 12:05:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4095 [2026-10-21 12:05:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4811 [2026-10-21 12:05:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4183 [2026-10-21 12:05:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6363 [2026-10-21 12:05:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4196 [2026-10-21 12:05:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1023 [2026-10-21 12:06:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4107 [2026-10-21 12:06:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5972 [2026-10-21 12:06:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6680 [2026-10-21 12:06:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6427 [2026-10-21 12:06:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8804 [2026-10-21 12:06:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4744 [2026-10-21 12:06:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.0248 [2026-10-21 12:06:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5379 [2026-10-21 12:06:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3778 [2026-10-21 12:06:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6361 [2026-10-21 12:06:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3889 [2026-10-21 12:06:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5537 [2026-10-21 12:07:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4714 [2026-10-21 12:07:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6047 [2026-10-21 12:07:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5029 [2026-10-21 12:07:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4959 [2026-10-21 12:07:16][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,609 | MFU: 1.8% [2026-10-21 12:07:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1362 [2026-10-21 12:07:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8381 [2026-10-21 12:07:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6104 [2026-10-21 12:07:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9585 [2026-10-21 12:07:21][INFO][training.pretrain_engine] [Ph2|Step 133] loss=5.2292 | ppl=186.65 | aux=0.02508 | lr=2.99e-04 | gnorm=0.641 | tok/s=29,390 | MFU=7.1% | step=142713ms | tokens=0.942B | mem=4.2/32.7GB | cost=$10.91/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 00:59 (132.9h) | Phase3: SKIPPED | SFT: Oct 27 05:43 (137.6h) | DPO (Final): Oct 27 06:17 (138.2h) [2026-10-21 12:07:21][INFO][training.pretrain_engine] Starting global step 134 (accumulating 32 micro-batches)... [2026-10-21 12:07:21][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 12:07:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8494 [2026-10-21 12:07:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9228 [2026-10-21 12:07:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7650 [2026-10-21 12:07:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5507 [2026-10-21 12:07:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6190 [2026-10-21 12:07:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5775 [2026-10-21 12:07:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3638 [2026-10-21 12:07:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5593 [2026-10-21 12:08:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5196 [2026-10-21 12:08:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4330 [2026-10-21 12:08:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7055 [2026-10-21 12:08:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3801 [2026-10-21 12:08:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1552 [2026-10-21 12:08:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1492 [2026-10-21 12:08:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3100 [2026-10-21 12:08:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0708 [2026-10-21 12:08:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0909 [2026-10-21 12:08:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9575 [2026-10-21 12:08:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2368 [2026-10-21 12:08:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1994 [2026-10-21 12:09:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6358 [2026-10-21 12:09:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8004 [2026-10-21 12:09:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8040 [2026-10-21 12:09:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8087 [2026-10-21 12:09:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1426 [2026-10-21 12:09:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6804 [2026-10-21 12:09:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5226 [2026-10-21 12:09:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3377 [2026-10-21 12:09:39][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,615 | MFU: 1.8% [2026-10-21 12:09:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4707 [2026-10-21 12:09:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9769 [2026-10-21 12:09:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6732 [2026-10-21 12:09:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4361 [2026-10-21 12:09:44][INFO][training.pretrain_engine] [Ph2|Step 134] loss=5.2417 | ppl=188.98 | aux=0.02506 | lr=2.99e-04 | gnorm=0.645 | tok/s=29,389 | MFU=7.1% | step=142718ms | tokens=0.943B | mem=4.2/32.7GB | cost=$10.99/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 01:01 (132.9h) | Phase3: SKIPPED | SFT: Oct 27 05:45 (137.6h) | DPO (Final): Oct 27 06:19 (138.2h) [2026-10-21 12:09:44][INFO][training.pretrain_engine] Starting global step 135 (accumulating 32 micro-batches)... [2026-10-21 12:09:44][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 12:10:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3620 [2026-10-21 12:10:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6541 [2026-10-21 12:10:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5087 [2026-10-21 12:10:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2269 [2026-10-21 12:10:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5104 [2026-10-21 12:10:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9040 [2026-10-21 12:10:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3652 [2026-10-21 12:10:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4004 [2026-10-21 12:10:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5707 [2026-10-21 12:10:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1653 [2026-10-21 12:10:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4480 [2026-10-21 12:10:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3877 [2026-10-21 12:10:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9134 [2026-10-21 12:10:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5653 [2026-10-21 12:10:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1856 [2026-10-21 12:10:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7612 [2026-10-21 12:11:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7125 [2026-10-21 12:11:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2259 [2026-10-21 12:11:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5365 [2026-10-21 12:11:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3850 [2026-10-21 12:11:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0424 [2026-10-21 12:11:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5182 [2026-10-21 12:11:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4371 [2026-10-21 12:11:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9875 [2026-10-21 12:11:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6181 [2026-10-21 12:11:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4170 [2026-10-21 12:11:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3138 [2026-10-21 12:11:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2946 [2026-10-21 12:12:02][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,622 | MFU: 1.8% [2026-10-21 12:12:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0338 [2026-10-21 12:12:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4157 [2026-10-21 12:12:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2334 [2026-10-21 12:12:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7767 [2026-10-21 12:12:08][INFO][training.pretrain_engine] [Ph2|Step 135] loss=5.2356 | ppl=187.83 | aux=0.02506 | lr=2.99e-04 | gnorm=0.746 | tok/s=29,387 | MFU=7.1% | step=142727ms | tokens=0.944B | mem=4.2/32.7GB | cost=$11.08/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 01:04 (132.9h) | Phase3: SKIPPED | SFT: Oct 27 05:47 (137.6h) | DPO (Final): Oct 27 06:21 (138.2h) [2026-10-21 12:12:11][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step135 | Tokens: 0.944B | Spent: $11.08 | Remaining: $988.92 [2026-10-21 12:12:11][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step135 [2026-10-21 12:12:11][INFO][training.pretrain_engine] Starting global step 136 (accumulating 32 micro-batches)... [2026-10-21 12:12:11][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 12:12:11][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 12:12:11][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 12:12:11][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 12:12:11][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step135 to Phase-Technologies/forge-500m-pretrain (path: phase2_step135)... [2026-10-21 12:12:11][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 12:12:25][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 12:12:25][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step135 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 12:12:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2155 [2026-10-21 12:12:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6763 [2026-10-21 12:12:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3027 [2026-10-21 12:12:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0763 [2026-10-21 12:12:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2856 [2026-10-21 12:12:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8823 [2026-10-21 12:12:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3073 [2026-10-21 12:12:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2949 [2026-10-21 12:13:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6085 [2026-10-21 12:13:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0045 [2026-10-21 12:13:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3487 [2026-10-21 12:13:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5004 [2026-10-21 12:13:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7506 [2026-10-21 12:13:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6414 [2026-10-21 12:13:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1184 [2026-10-21 12:13:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2004 [2026-10-21 12:13:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5010 [2026-10-21 12:13:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5594 [2026-10-21 12:13:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7325 [2026-10-21 12:13:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4974 [2026-10-21 12:13:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6772 [2026-10-21 12:13:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.0890 [2026-10-21 12:13:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2202 [2026-10-21 12:13:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3939 [2026-10-21 12:14:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7806 [2026-10-21 12:14:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9010 [2026-10-21 12:14:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6803 [2026-10-21 12:14:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8151 [2026-10-21 12:14:28][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,612 | MFU: 1.8% [2026-10-21 12:14:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5855 [2026-10-21 12:14:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2578 [2026-10-21 12:14:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1856 [2026-10-21 12:14:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0853 [2026-10-21 12:14:33][INFO][training.pretrain_engine] [Ph2|Step 136] loss=5.2647 | ppl=193.39 | aux=0.02507 | lr=2.99e-04 | gnorm=0.594 | tok/s=29,387 | MFU=7.1% | step=142726ms | tokens=0.945B | mem=4.2/32.7GB | cost=$11.16/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 01:05 (132.9h) | Phase3: SKIPPED | SFT: Oct 27 05:49 (137.6h) | DPO (Final): Oct 27 06:23 (138.1h) [2026-10-21 12:14:33][INFO][training.pretrain_engine] Starting global step 137 (accumulating 32 micro-batches)... [2026-10-21 12:14:33][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 12:14:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1668 [2026-10-21 12:14:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5317 [2026-10-21 12:14:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3218 [2026-10-21 12:14:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7611 [2026-10-21 12:15:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9969 [2026-10-21 12:15:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3270 [2026-10-21 12:15:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7850 [2026-10-21 12:15:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1217 [2026-10-21 12:15:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0768 [2026-10-21 12:15:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3344 [2026-10-21 12:15:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5275 [2026-10-21 12:15:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5004 [2026-10-21 12:15:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2807 [2026-10-21 12:15:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0362 [2026-10-21 12:15:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1843 [2026-10-21 12:15:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2453 [2026-10-21 12:16:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8195 [2026-10-21 12:16:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7860 [2026-10-21 12:16:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8134 [2026-10-21 12:16:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9805 [2026-10-21 12:16:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4229 [2026-10-21 12:16:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4747 [2026-10-21 12:16:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3017 [2026-10-21 12:16:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2447 [2026-10-21 12:16:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6096 [2026-10-21 12:16:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2884 [2026-10-21 12:16:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9856 [2026-10-21 12:16:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4114 [2026-10-21 12:16:51][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,604 | MFU: 1.8% [2026-10-21 12:16:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8689 [2026-10-21 12:16:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8474 [2026-10-21 12:16:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4611 [2026-10-21 12:16:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7581 [2026-10-21 12:16:56][INFO][training.pretrain_engine] [Ph2|Step 137] loss=5.2796 | ppl=196.29 | aux=0.02506 | lr=2.99e-04 | gnorm=0.742 | tok/s=29,385 | MFU=7.1% | step=142736ms | tokens=0.946B | mem=4.2/32.7GB | cost=$11.24/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 01:08 (132.9h) | Phase3: SKIPPED | SFT: Oct 27 05:51 (137.6h) | DPO (Final): Oct 27 06:25 (138.1h) [2026-10-21 12:16:56][INFO][training.pretrain_engine] Starting global step 138 (accumulating 32 micro-batches)... [2026-10-21 12:16:56][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 12:17:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0459 [2026-10-21 12:17:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3278 [2026-10-21 12:17:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8652 [2026-10-21 12:17:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3871 [2026-10-21 12:17:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2879 [2026-10-21 12:17:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6374 [2026-10-21 12:17:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0498 [2026-10-21 12:17:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0763 [2026-10-21 12:17:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6219 [2026-10-21 12:17:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5986 [2026-10-21 12:17:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5916 [2026-10-21 12:17:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4152 [2026-10-21 12:18:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9732 [2026-10-21 12:18:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1291 [2026-10-21 12:18:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2007 [2026-10-21 12:18:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9932 [2026-10-21 12:18:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3156 [2026-10-21 12:18:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7697 [2026-10-21 12:18:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9646 [2026-10-21 12:18:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2387 [2026-10-21 12:18:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6361 [2026-10-21 12:18:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4496 [2026-10-21 12:18:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8809 [2026-10-21 12:18:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6820 [2026-10-21 12:19:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8650 [2026-10-21 12:19:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7662 [2026-10-21 12:19:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7376 [2026-10-21 12:19:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3290 [2026-10-21 12:19:13][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,626 | MFU: 1.8% [2026-10-21 12:19:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9615 [2026-10-21 12:19:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7845 [2026-10-21 12:19:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9451 [2026-10-21 12:19:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6151 [2026-10-21 12:19:19][INFO][training.pretrain_engine] [Ph2|Step 138] loss=5.1988 | ppl=181.05 | aux=0.02509 | lr=2.99e-04 | gnorm=0.598 | tok/s=29,385 | MFU=7.1% | step=142738ms | tokens=0.947B | mem=4.2/32.7GB | cost=$11.32/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 01:10 (132.8h) | Phase3: SKIPPED | SFT: Oct 27 05:53 (137.6h) | DPO (Final): Oct 27 06:27 (138.1h) [2026-10-21 12:19:19][INFO][training.pretrain_engine] Starting global step 139 (accumulating 32 micro-batches)... [2026-10-21 12:19:19][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 12:19:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4574 [2026-10-21 12:19:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9306 [2026-10-21 12:19:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4405 [2026-10-21 12:19:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9396 [2026-10-21 12:19:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8661 [2026-10-21 12:19:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4975 [2026-10-21 12:19:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2024 [2026-10-21 12:19:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1630 [2026-10-21 12:20:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9586 [2026-10-21 12:20:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6045 [2026-10-21 12:20:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0460 [2026-10-21 12:20:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2308 [2026-10-21 12:20:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0694 [2026-10-21 12:20:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0618 [2026-10-21 12:20:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2068 [2026-10-21 12:20:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8683 [2026-10-21 12:20:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9715 [2026-10-21 12:20:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7409 [2026-10-21 12:20:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.0886 [2026-10-21 12:20:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.1847 [2026-10-21 12:21:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7354 [2026-10-21 12:21:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8312 [2026-10-21 12:21:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9615 [2026-10-21 12:21:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1422 [2026-10-21 12:21:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1061 [2026-10-21 12:21:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0892 [2026-10-21 12:21:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1104 [2026-10-21 12:21:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2515 [2026-10-21 12:21:36][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,621 | MFU: 1.8% [2026-10-21 12:21:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2380 [2026-10-21 12:21:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4387 [2026-10-21 12:21:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9895 [2026-10-21 12:21:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2154 [2026-10-21 12:21:41][INFO][training.pretrain_engine] [Ph2|Step 139] loss=5.2896 | ppl=198.27 | aux=0.02507 | lr=2.99e-04 | gnorm=0.660 | tok/s=29,384 | MFU=7.1% | step=142741ms | tokens=0.948B | mem=4.2/32.7GB | cost=$11.40/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 01:12 (132.8h) | Phase3: SKIPPED | SFT: Oct 27 05:55 (137.6h) | DPO (Final): Oct 27 06:29 (138.1h) [2026-10-21 12:21:41][INFO][training.pretrain_engine] Starting global step 140 (accumulating 32 micro-batches)... [2026-10-21 12:21:41][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 12:21:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3295 [2026-10-21 12:21:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2900 [2026-10-21 12:21:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1266 [2026-10-21 12:21:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7540 [2026-10-21 12:22:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6182 [2026-10-21 12:22:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1699 [2026-10-21 12:22:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5498 [2026-10-21 12:22:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5638 [2026-10-21 12:22:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0434 [2026-10-21 12:22:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1240 [2026-10-21 12:22:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7583 [2026-10-21 12:22:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3137 [2026-10-21 12:22:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1315 [2026-10-21 12:22:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1519 [2026-10-21 12:22:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7925 [2026-10-21 12:22:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9611 [2026-10-21 12:23:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0421 [2026-10-21 12:23:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0391 [2026-10-21 12:23:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8579 [2026-10-21 12:23:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4446 [2026-10-21 12:23:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4210 [2026-10-21 12:23:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4891 [2026-10-21 12:23:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3174 [2026-10-21 12:23:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4183 [2026-10-21 12:23:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3651 [2026-10-21 12:23:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7306 [2026-10-21 12:23:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1085 [2026-10-21 12:23:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2862 [2026-10-21 12:23:59][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,603 | MFU: 1.8% [2026-10-21 12:24:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6225 [2026-10-21 12:24:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4031 [2026-10-21 12:24:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4890 [2026-10-21 12:24:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6246 [2026-10-21 12:24:04][INFO][training.pretrain_engine] [Ph2|Step 140] loss=5.2926 | ppl=198.86 | aux=0.02506 | lr=2.99e-04 | gnorm=0.482 | tok/s=29,383 | MFU=7.1% | step=142747ms | tokens=0.949B | mem=4.2/32.7GB | cost=$11.49/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 01:14 (132.8h) | Phase3: SKIPPED | SFT: Oct 27 05:57 (137.6h) | DPO (Final): Oct 27 06:31 (138.1h) [2026-10-21 12:24:04][INFO][training.pretrain_engine] Starting global step 141 (accumulating 32 micro-batches)... [2026-10-21 12:24:04][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 12:24:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8061 [2026-10-21 12:24:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7943 [2026-10-21 12:24:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1010 [2026-10-21 12:24:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8126 [2026-10-21 12:24:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2362 [2026-10-21 12:24:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4755 [2026-10-21 12:24:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6379 [2026-10-21 12:24:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1328 [2026-10-21 12:24:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5948 [2026-10-21 12:24:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3051 [2026-10-21 12:24:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0398 [2026-10-21 12:24:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6733 [2026-10-21 12:25:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4139 [2026-10-21 12:25:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0885 [2026-10-21 12:25:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0273 [2026-10-21 12:25:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8646 [2026-10-21 12:25:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1611 [2026-10-21 12:25:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7972 [2026-10-21 12:25:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9342 [2026-10-21 12:25:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0252 [2026-10-21 12:25:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2695 [2026-10-21 12:25:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2072 [2026-10-21 12:25:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0826 [2026-10-21 12:25:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5871 [2026-10-21 12:26:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2426 [2026-10-21 12:26:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2283 [2026-10-21 12:26:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1346 [2026-10-21 12:26:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4187 [2026-10-21 12:26:21][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,623 | MFU: 1.8% [2026-10-21 12:26:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2757 [2026-10-21 12:26:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8568 [2026-10-21 12:26:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3121 [2026-10-21 12:26:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5161 [2026-10-21 12:26:26][INFO][training.pretrain_engine] [Ph2|Step 141] loss=5.1973 | ppl=180.78 | aux=0.02506 | lr=2.99e-04 | gnorm=0.676 | tok/s=29,382 | MFU=7.1% | step=142749ms | tokens=0.950B | mem=4.2/32.7GB | cost=$11.57/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 01:16 (132.8h) | Phase3: SKIPPED | SFT: Oct 27 05:59 (137.6h) | DPO (Final): Oct 27 06:33 (138.1h) [2026-10-21 12:26:26][INFO][training.pretrain_engine] Starting global step 142 (accumulating 32 micro-batches)... [2026-10-21 12:26:26][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 12:26:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7120 [2026-10-21 12:26:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3836 [2026-10-21 12:26:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5165 [2026-10-21 12:26:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6311 [2026-10-21 12:27:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4082 [2026-10-21 12:27:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9072 [2026-10-21 12:27:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2501 [2026-10-21 12:27:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2397 [2026-10-21 12:27:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3541 [2026-10-21 12:27:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2267 [2026-10-21 12:27:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3498 [2026-10-21 12:27:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9992 [2026-10-21 12:27:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2852 [2026-10-21 12:27:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2882 [2026-10-21 12:27:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9557 [2026-10-21 12:27:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1063 [2026-10-21 12:27:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6110 [2026-10-21 12:27:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3919 [2026-10-21 12:27:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6682 [2026-10-21 12:27:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0432 [2026-10-21 12:28:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3234 [2026-10-21 12:28:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2309 [2026-10-21 12:28:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9294 [2026-10-21 12:28:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2829 [2026-10-21 12:28:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5033 [2026-10-21 12:28:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4841 [2026-10-21 12:28:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3177 [2026-10-21 12:28:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4539 [2026-10-21 12:28:44][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,605 | MFU: 1.8% [2026-10-21 12:28:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0376 [2026-10-21 12:28:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9745 [2026-10-21 12:28:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8181 [2026-10-21 12:28:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1067 [2026-10-21 12:28:49][INFO][training.pretrain_engine] [Ph2|Step 142] loss=5.2277 | ppl=186.36 | aux=0.02506 | lr=2.99e-04 | gnorm=0.594 | tok/s=29,381 | MFU=7.1% | step=142756ms | tokens=0.951B | mem=4.2/32.7GB | cost=$11.65/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 01:18 (132.8h) | Phase3: SKIPPED | SFT: Oct 27 06:01 (137.6h) | DPO (Final): Oct 27 06:35 (138.1h) [2026-10-21 12:28:49][INFO][training.pretrain_engine] Starting global step 143 (accumulating 32 micro-batches)... [2026-10-21 12:28:49][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 12:29:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3512 [2026-10-21 12:29:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6084 [2026-10-21 12:29:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7958 [2026-10-21 12:29:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6342 [2026-10-21 12:29:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1409 [2026-10-21 12:29:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2755 [2026-10-21 12:29:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3125 [2026-10-21 12:29:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0261 [2026-10-21 12:29:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4943 [2026-10-21 12:29:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1523 [2026-10-21 12:29:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2406 [2026-10-21 12:29:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1689 [2026-10-21 12:30:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3825 [2026-10-21 12:30:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1765 [2026-10-21 12:30:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7000 [2026-10-21 12:30:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2715 [2026-10-21 12:30:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0234 [2026-10-21 12:30:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8665 [2026-10-21 12:30:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0051 [2026-10-21 12:30:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2022 [2026-10-21 12:30:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2498 [2026-10-21 12:30:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2445 [2026-10-21 12:30:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6301 [2026-10-21 12:30:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3583 [2026-10-21 12:30:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2682 [2026-10-21 12:30:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2881 [2026-10-21 12:30:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3100 [2026-10-21 12:30:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0934 [2026-10-21 12:31:07][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,619 | MFU: 1.8% [2026-10-21 12:31:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2337 [2026-10-21 12:31:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3019 [2026-10-21 12:31:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8461 [2026-10-21 12:31:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3575 [2026-10-21 12:31:11][INFO][training.pretrain_engine] [Ph2|Step 143] loss=5.3358 | ppl=207.63 | aux=0.02506 | lr=2.99e-04 | gnorm=0.605 | tok/s=29,380 | MFU=7.1% | step=142760ms | tokens=0.952B | mem=4.2/32.7GB | cost=$11.73/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 01:20 (132.8h) | Phase3: SKIPPED | SFT: Oct 27 06:03 (137.5h) | DPO (Final): Oct 27 06:37 (138.1h) [2026-10-21 12:31:11][INFO][training.pretrain_engine] Starting global step 144 (accumulating 32 micro-batches)... [2026-10-21 12:31:11][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 12:31:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1923 [2026-10-21 12:31:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5240 [2026-10-21 12:31:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8097 [2026-10-21 12:31:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1974 [2026-10-21 12:31:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3502 [2026-10-21 12:31:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5642 [2026-10-21 12:31:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0247 [2026-10-21 12:31:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5069 [2026-10-21 12:32:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3848 [2026-10-21 12:32:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9236 [2026-10-21 12:32:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9205 [2026-10-21 12:32:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3554 [2026-10-21 12:32:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3994 [2026-10-21 12:32:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1088 [2026-10-21 12:32:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7583 [2026-10-21 12:32:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5308 [2026-10-21 12:32:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3721 [2026-10-21 12:32:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1323 [2026-10-21 12:32:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3786 [2026-10-21 12:32:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1133 [2026-10-21 12:32:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3732 [2026-10-21 12:32:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5498 [2026-10-21 12:32:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3681 [2026-10-21 12:32:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8011 [2026-10-21 12:33:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5325 [2026-10-21 12:33:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4948 [2026-10-21 12:33:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6385 [2026-10-21 12:33:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4459 [2026-10-21 12:33:29][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,617 | MFU: 1.8% [2026-10-21 12:33:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2072 [2026-10-21 12:33:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9542 [2026-10-21 12:33:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0848 [2026-10-21 12:33:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4927 [2026-10-21 12:33:34][INFO][training.pretrain_engine] [Ph2|Step 144] loss=5.3040 | ppl=201.13 | aux=0.02507 | lr=2.99e-04 | gnorm=0.492 | tok/s=29,381 | MFU=7.1% | step=142757ms | tokens=0.953B | mem=4.2/32.7GB | cost=$11.81/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 01:21 (132.8h) | Phase3: SKIPPED | SFT: Oct 27 06:05 (137.5h) | DPO (Final): Oct 27 06:39 (138.1h) [2026-10-21 12:33:37][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step144 | Tokens: 0.953B | Spent: $11.81 | Remaining: $988.19 [2026-10-21 12:33:37][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step144 [2026-10-21 12:33:37][INFO][training.pretrain_engine] Starting global step 145 (accumulating 32 micro-batches)... [2026-10-21 12:33:37][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 12:33:37][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 12:33:37][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 12:33:37][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 12:33:37][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step144 to Phase-Technologies/forge-500m-pretrain (path: phase2_step144)... [2026-10-21 12:33:37][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 12:33:49][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 12:33:49][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step144 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 12:33:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0179 [2026-10-21 12:33:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4055 [2026-10-21 12:33:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2516 [2026-10-21 12:33:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0645 [2026-10-21 12:34:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8993 [2026-10-21 12:34:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7532 [2026-10-21 12:34:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7155 [2026-10-21 12:34:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7689 [2026-10-21 12:34:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2655 [2026-10-21 12:34:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3130 [2026-10-21 12:34:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1831 [2026-10-21 12:34:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9324 [2026-10-21 12:34:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5735 [2026-10-21 12:34:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1654 [2026-10-21 12:34:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1384 [2026-10-21 12:34:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1284 [2026-10-21 12:35:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2061 [2026-10-21 12:35:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8272 [2026-10-21 12:35:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5537 [2026-10-21 12:35:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7177 [2026-10-21 12:35:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8308 [2026-10-21 12:35:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2298 [2026-10-21 12:35:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5566 [2026-10-21 12:35:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5740 [2026-10-21 12:35:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5788 [2026-10-21 12:35:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3065 [2026-10-21 12:35:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8496 [2026-10-21 12:35:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7841 [2026-10-21 12:35:54][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,610 | MFU: 1.8% [2026-10-21 12:35:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0527 [2026-10-21 12:35:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2043 [2026-10-21 12:35:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2091 [2026-10-21 12:35:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1894 [2026-10-21 12:35:59][INFO][training.pretrain_engine] [Ph2|Step 145] loss=5.2991 | ppl=200.15 | aux=0.02506 | lr=2.99e-04 | gnorm=0.547 | tok/s=29,381 | MFU=7.1% | step=142755ms | tokens=0.954B | mem=4.2/32.7GB | cost=$11.90/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 01:23 (132.8h) | Phase3: SKIPPED | SFT: Oct 27 06:07 (137.5h) | DPO (Final): Oct 27 06:41 (138.1h) [2026-10-21 12:35:59][INFO][training.pretrain_engine] Starting global step 146 (accumulating 32 micro-batches)... [2026-10-21 12:35:59][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 12:36:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5107 [2026-10-21 12:36:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8231 [2026-10-21 12:36:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6462 [2026-10-21 12:36:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6956 [2026-10-21 12:36:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4361 [2026-10-21 12:36:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2190 [2026-10-21 12:36:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2530 [2026-10-21 12:36:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6087 [2026-10-21 12:36:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9622 [2026-10-21 12:36:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7992 [2026-10-21 12:36:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4533 [2026-10-21 12:36:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0987 [2026-10-21 12:37:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1281 [2026-10-21 12:37:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6199 [2026-10-21 12:37:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2767 [2026-10-21 12:37:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9538 [2026-10-21 12:37:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1774 [2026-10-21 12:37:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8868 [2026-10-21 12:37:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2872 [2026-10-21 12:37:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2024 [2026-10-21 12:37:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0768 [2026-10-21 12:37:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4676 [2026-10-21 12:37:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4742 [2026-10-21 12:37:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4193 [2026-10-21 12:38:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1885 [2026-10-21 12:38:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3974 [2026-10-21 12:38:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2901 [2026-10-21 12:38:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4134 [2026-10-21 12:38:17][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,609 | MFU: 1.8% [2026-10-21 12:38:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7288 [2026-10-21 12:38:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9938 [2026-10-21 12:38:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7114 [2026-10-21 12:38:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5373 [2026-10-21 12:38:21][INFO][training.pretrain_engine] [Ph2|Step 146] loss=5.3235 | ppl=205.11 | aux=0.02506 | lr=2.99e-04 | gnorm=0.531 | tok/s=29,380 | MFU=7.1% | step=142761ms | tokens=0.955B | mem=4.2/32.7GB | cost=$11.98/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 01:25 (132.8h) | Phase3: SKIPPED | SFT: Oct 27 06:09 (137.5h) | DPO (Final): Oct 27 06:43 (138.1h) [2026-10-21 12:38:21][INFO][training.pretrain_engine] Starting global step 147 (accumulating 32 micro-batches)... [2026-10-21 12:38:21][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 12:38:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4123 [2026-10-21 12:38:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2749 [2026-10-21 12:38:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2506 [2026-10-21 12:38:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2820 [2026-10-21 12:38:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0028 [2026-10-21 12:38:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3530 [2026-10-21 12:38:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9977 [2026-10-21 12:38:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0920 [2026-10-21 12:39:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4215 [2026-10-21 12:39:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6766 [2026-10-21 12:39:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5118 [2026-10-21 12:39:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7240 [2026-10-21 12:39:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5387 [2026-10-21 12:39:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8837 [2026-10-21 12:39:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0607 [2026-10-21 12:39:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5898 [2026-10-21 12:39:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4701 [2026-10-21 12:39:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2904 [2026-10-21 12:39:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0799 [2026-10-21 12:39:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2201 [2026-10-21 12:40:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1399 [2026-10-21 12:40:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3396 [2026-10-21 12:40:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2676 [2026-10-21 12:40:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5412 [2026-10-21 12:40:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8426 [2026-10-21 12:40:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2004 [2026-10-21 12:40:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5375 [2026-10-21 12:40:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6735 [2026-10-21 12:40:39][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,635 | MFU: 1.8% [2026-10-21 12:40:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7558 [2026-10-21 12:40:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8837 [2026-10-21 12:40:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7567 [2026-10-21 12:40:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6490 [2026-10-21 12:40:44][INFO][training.pretrain_engine] [Ph2|Step 147] loss=5.3017 | ppl=200.68 | aux=0.02506 | lr=2.99e-04 | gnorm=0.605 | tok/s=29,380 | MFU=7.1% | step=142760ms | tokens=0.956B | mem=4.2/32.7GB | cost=$12.06/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 01:27 (132.8h) | Phase3: SKIPPED | SFT: Oct 27 06:11 (137.5h) | DPO (Final): Oct 27 06:45 (138.1h) [2026-10-21 12:40:44][INFO][training.pretrain_engine] Starting global step 148 (accumulating 32 micro-batches)... [2026-10-21 12:40:44][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 12:41:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8867 [2026-10-21 12:41:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3640 [2026-10-21 12:41:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4733 [2026-10-21 12:41:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4230 [2026-10-21 12:41:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3847 [2026-10-21 12:41:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5106 [2026-10-21 12:41:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1757 [2026-10-21 12:41:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3448 [2026-10-21 12:41:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3711 [2026-10-21 12:41:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0655 [2026-10-21 12:41:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0444 [2026-10-21 12:41:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9220 [2026-10-21 12:41:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7012 [2026-10-21 12:41:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6872 [2026-10-21 12:41:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9488 [2026-10-21 12:41:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7965 [2026-10-21 12:42:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3458 [2026-10-21 12:42:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2781 [2026-10-21 12:42:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3247 [2026-10-21 12:42:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2294 [2026-10-21 12:42:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2701 [2026-10-21 12:42:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2404 [2026-10-21 12:42:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7337 [2026-10-21 12:42:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3691 [2026-10-21 12:42:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1435 [2026-10-21 12:42:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5602 [2026-10-21 12:42:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3499 [2026-10-21 12:42:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6352 [2026-10-21 12:43:01][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,635 | MFU: 1.8% [2026-10-21 12:43:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7201 [2026-10-21 12:43:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8812 [2026-10-21 12:43:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2950 [2026-10-21 12:43:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3016 [2026-10-21 12:43:07][INFO][training.pretrain_engine] [Ph2|Step 148] loss=5.3294 | ppl=206.31 | aux=0.02506 | lr=2.99e-04 | gnorm=0.617 | tok/s=29,380 | MFU=7.1% | step=142759ms | tokens=0.957B | mem=4.2/32.7GB | cost=$12.14/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 01:29 (132.8h) | Phase3: SKIPPED | SFT: Oct 27 06:12 (137.5h) | DPO (Final): Oct 27 06:46 (138.1h) [2026-10-21 12:43:07][INFO][training.pretrain_engine] Starting global step 149 (accumulating 32 micro-batches)... [2026-10-21 12:43:07][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 12:43:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1100 [2026-10-21 12:43:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9802 [2026-10-21 12:43:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4074 [2026-10-21 12:43:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0913 [2026-10-21 12:43:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4557 [2026-10-21 12:43:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1240 [2026-10-21 12:43:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8618 [2026-10-21 12:43:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4526 [2026-10-21 12:44:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5845 [2026-10-21 12:44:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4655 [2026-10-21 12:44:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3328 [2026-10-21 12:44:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6288 [2026-10-21 12:44:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8077 [2026-10-21 12:44:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4260 [2026-10-21 12:44:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3790 [2026-10-21 12:44:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5371 [2026-10-21 12:44:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2679 [2026-10-21 12:44:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1360 [2026-10-21 12:44:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9864 [2026-10-21 12:44:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2264 [2026-10-21 12:44:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7755 [2026-10-21 12:44:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3875 [2026-10-21 12:44:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5511 [2026-10-21 12:44:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4037 [2026-10-21 12:45:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2807 [2026-10-21 12:45:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4178 [2026-10-21 12:45:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5111 [2026-10-21 12:45:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0434 [2026-10-21 12:45:25][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,609 | MFU: 1.8% [2026-10-21 12:45:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7462 [2026-10-21 12:45:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6167 [2026-10-21 12:45:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8977 [2026-10-21 12:45:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8060 [2026-10-21 12:45:30][INFO][training.pretrain_engine] [Ph2|Step 149] loss=5.1478 | ppl=172.04 | aux=0.02509 | lr=2.99e-04 | gnorm=0.871 | tok/s=29,381 | MFU=7.1% | step=142757ms | tokens=0.958B | mem=4.2/32.7GB | cost=$12.22/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 01:30 (132.8h) | Phase3: SKIPPED | SFT: Oct 27 06:14 (137.5h) | DPO (Final): Oct 27 06:48 (138.0h) [2026-10-21 12:45:30][INFO][training.pretrain_engine] Starting global step 150 (accumulating 32 micro-batches)... [2026-10-21 12:45:30][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 12:45:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4902 [2026-10-21 12:45:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8062 [2026-10-21 12:45:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8582 [2026-10-21 12:45:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0821 [2026-10-21 12:46:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1991 [2026-10-21 12:46:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1934 [2026-10-21 12:46:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2244 [2026-10-21 12:46:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2752 [2026-10-21 12:46:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1120 [2026-10-21 12:46:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0359 [2026-10-21 12:46:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9322 [2026-10-21 12:46:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1732 [2026-10-21 12:46:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2420 [2026-10-21 12:46:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7787 [2026-10-21 12:46:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=7.0137 [2026-10-21 12:46:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9914 [2026-10-21 12:46:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4783 [2026-10-21 12:46:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0504 [2026-10-21 12:46:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1529 [2026-10-21 12:46:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3392 [2026-10-21 12:47:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3303 [2026-10-21 12:47:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3128 [2026-10-21 12:47:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2783 [2026-10-21 12:47:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3593 [2026-10-21 12:47:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6844 [2026-10-21 12:47:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2165 [2026-10-21 12:47:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5455 [2026-10-21 12:47:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9610 [2026-10-21 12:47:48][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,602 | MFU: 1.8% [2026-10-21 12:47:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8380 [2026-10-21 12:47:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7426 [2026-10-21 12:47:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1134 [2026-10-21 12:47:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5268 [2026-10-21 12:47:53][INFO][training.pretrain_engine] [Ph2|Step 150] loss=5.2465 | ppl=189.90 | aux=0.02508 | lr=2.99e-04 | gnorm=1.328 | tok/s=29,378 | MFU=7.1% | step=142769ms | tokens=0.959B | mem=4.2/32.7GB | cost=$12.30/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 01:33 (132.8h) | Phase3: SKIPPED | SFT: Oct 27 06:16 (137.5h) | DPO (Final): Oct 27 06:50 (138.1h) [2026-10-21 12:47:53][INFO][training.pretrain_engine] Starting global step 151 (accumulating 32 micro-batches)... [2026-10-21 12:47:53][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 12:48:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9298 [2026-10-21 12:48:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3012 [2026-10-21 12:48:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9832 [2026-10-21 12:48:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0061 [2026-10-21 12:48:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0938 [2026-10-21 12:48:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5184 [2026-10-21 12:48:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6677 [2026-10-21 12:48:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3562 [2026-10-21 12:48:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9384 [2026-10-21 12:48:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1292 [2026-10-21 12:48:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1775 [2026-10-21 12:48:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9334 [2026-10-21 12:49:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5696 [2026-10-21 12:49:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0695 [2026-10-21 12:49:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4624 [2026-10-21 12:49:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5879 [2026-10-21 12:49:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8421 [2026-10-21 12:49:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6249 [2026-10-21 12:49:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4500 [2026-10-21 12:49:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0110 [2026-10-21 12:49:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8981 [2026-10-21 12:49:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8053 [2026-10-21 12:49:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5684 [2026-10-21 12:49:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5556 [2026-10-21 12:49:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1672 [2026-10-21 12:49:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1869 [2026-10-21 12:49:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0838 [2026-10-21 12:49:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8997 [2026-10-21 12:50:11][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,612 | MFU: 1.8% [2026-10-21 12:50:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6551 [2026-10-21 12:50:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9767 [2026-10-21 12:50:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4689 [2026-10-21 12:50:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3646 [2026-10-21 12:50:16][INFO][training.pretrain_engine] [Ph2|Step 151] loss=5.2090 | ppl=182.90 | aux=0.02507 | lr=2.99e-04 | gnorm=0.926 | tok/s=29,377 | MFU=7.1% | step=142774ms | tokens=0.960B | mem=4.2/32.7GB | cost=$12.39/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 01:35 (132.8h) | Phase3: SKIPPED | SFT: Oct 27 06:18 (137.5h) | DPO (Final): Oct 27 06:53 (138.0h) [2026-10-21 12:50:16][INFO][training.pretrain_engine] Starting global step 152 (accumulating 32 micro-batches)... [2026-10-21 12:50:16][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 12:50:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4717 [2026-10-21 12:50:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1796 [2026-10-21 12:50:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1362 [2026-10-21 12:50:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4045 [2026-10-21 12:50:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3613 [2026-10-21 12:50:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4520 [2026-10-21 12:50:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8036 [2026-10-21 12:50:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1434 [2026-10-21 12:51:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8144 [2026-10-21 12:51:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5580 [2026-10-21 12:51:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9612 [2026-10-21 12:51:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9249 [2026-10-21 12:51:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6977 [2026-10-21 12:51:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0317 [2026-10-21 12:51:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0361 [2026-10-21 12:51:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5284 [2026-10-21 12:51:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4635 [2026-10-21 12:51:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5055 [2026-10-21 12:51:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6883 [2026-10-21 12:51:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9261 [2026-10-21 12:52:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9517 [2026-10-21 12:52:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2187 [2026-10-21 12:52:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2437 [2026-10-21 12:52:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0113 [2026-10-21 12:52:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2931 [2026-10-21 12:52:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1956 [2026-10-21 12:52:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4952 [2026-10-21 12:52:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6841 [2026-10-21 12:52:34][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,607 | MFU: 1.8% [2026-10-21 12:52:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7553 [2026-10-21 12:52:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9185 [2026-10-21 12:52:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8025 [2026-10-21 12:52:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0591 [2026-10-21 12:52:39][INFO][training.pretrain_engine] [Ph2|Step 152] loss=5.2367 | ppl=188.05 | aux=0.02507 | lr=2.99e-04 | gnorm=0.695 | tok/s=29,377 | MFU=7.1% | step=142773ms | tokens=0.962B | mem=4.2/32.7GB | cost=$12.47/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 01:37 (132.7h) | Phase3: SKIPPED | SFT: Oct 27 06:20 (137.5h) | DPO (Final): Oct 27 06:54 (138.0h) [2026-10-21 12:52:39][INFO][training.pretrain_engine] Starting global step 153 (accumulating 32 micro-batches)... [2026-10-21 12:52:39][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 12:52:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3149 [2026-10-21 12:52:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5502 [2026-10-21 12:52:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3595 [2026-10-21 12:52:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4633 [2026-10-21 12:53:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2789 [2026-10-21 12:53:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1261 [2026-10-21 12:53:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2558 [2026-10-21 12:53:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1360 [2026-10-21 12:53:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4830 [2026-10-21 12:53:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8860 [2026-10-21 12:53:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7975 [2026-10-21 12:53:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5675 [2026-10-21 12:53:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1767 [2026-10-21 12:53:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3332 [2026-10-21 12:53:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2262 [2026-10-21 12:53:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2783 [2026-10-21 12:54:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4930 [2026-10-21 12:54:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4293 [2026-10-21 12:54:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1241 [2026-10-21 12:54:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3181 [2026-10-21 12:54:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0203 [2026-10-21 12:54:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3331 [2026-10-21 12:54:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9993 [2026-10-21 12:54:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6546 [2026-10-21 12:54:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7390 [2026-10-21 12:54:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5501 [2026-10-21 12:54:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5979 [2026-10-21 12:54:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3199 [2026-10-21 12:54:57][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,605 | MFU: 1.8% [2026-10-21 12:55:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1835 [2026-10-21 12:55:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4677 [2026-10-21 12:55:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0073 [2026-10-21 12:55:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=3.5597 [2026-10-21 12:55:01][INFO][training.pretrain_engine] [Ph2|Step 153] loss=5.1439 | ppl=171.38 | aux=0.02506 | lr=2.99e-04 | gnorm=0.738 | tok/s=29,377 | MFU=7.1% | step=142774ms | tokens=0.963B | mem=4.2/32.7GB | cost=$12.55/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 01:38 (132.7h) | Phase3: SKIPPED | SFT: Oct 27 06:22 (137.5h) | DPO (Final): Oct 27 06:56 (138.0h) [2026-10-21 12:55:05][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step153 | Tokens: 0.963B | Spent: $12.55 | Remaining: $987.45 [2026-10-21 12:55:05][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step153 [2026-10-21 12:55:05][INFO][training.pretrain_engine] Starting global step 154 (accumulating 32 micro-batches)... [2026-10-21 12:55:05][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 12:55:05][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 12:55:05][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 12:55:05][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 12:55:05][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step153 to Phase-Technologies/forge-500m-pretrain (path: phase2_step153)... [2026-10-21 12:55:05][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 12:55:14][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 12:55:14][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step153 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 12:55:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7593 [2026-10-21 12:55:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2769 [2026-10-21 12:55:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5962 [2026-10-21 12:55:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7490 [2026-10-21 12:55:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5975 [2026-10-21 12:55:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5360 [2026-10-21 12:55:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3881 [2026-10-21 12:55:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3432 [2026-10-21 12:55:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8433 [2026-10-21 12:55:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2017 [2026-10-21 12:55:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1772 [2026-10-21 12:55:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0858 [2026-10-21 12:56:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9945 [2026-10-21 12:56:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1634 [2026-10-21 12:56:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8174 [2026-10-21 12:56:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1386 [2026-10-21 12:56:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1116 [2026-10-21 12:56:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6933 [2026-10-21 12:56:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7954 [2026-10-21 12:56:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2115 [2026-10-21 12:56:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9191 [2026-10-21 12:56:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4077 [2026-10-21 12:56:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0144 [2026-10-21 12:56:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2467 [2026-10-21 12:57:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4154 [2026-10-21 12:57:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5479 [2026-10-21 12:57:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2287 [2026-10-21 12:57:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7308 [2026-10-21 12:57:22][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,623 | MFU: 1.8% [2026-10-21 12:57:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2389 [2026-10-21 12:57:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1643 [2026-10-21 12:57:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1356 [2026-10-21 12:57:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2067 [2026-10-21 12:57:27][INFO][training.pretrain_engine] [Ph2|Step 154] loss=5.3085 | ppl=202.04 | aux=0.02507 | lr=2.99e-04 | gnorm=0.723 | tok/s=29,378 | MFU=7.1% | step=142771ms | tokens=0.964B | mem=4.2/32.7GB | cost=$12.63/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 01:40 (132.7h) | Phase3: SKIPPED | SFT: Oct 27 06:24 (137.4h) | DPO (Final): Oct 27 06:58 (138.0h) [2026-10-21 12:57:27][INFO][training.pretrain_engine] Starting global step 155 (accumulating 32 micro-batches)... [2026-10-21 12:57:27][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 12:57:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7133 [2026-10-21 12:57:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2383 [2026-10-21 12:57:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5013 [2026-10-21 12:57:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7498 [2026-10-21 12:58:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7368 [2026-10-21 12:58:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1225 [2026-10-21 12:58:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2482 [2026-10-21 12:58:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9491 [2026-10-21 12:58:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3855 [2026-10-21 12:58:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7623 [2026-10-21 12:58:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3881 [2026-10-21 12:58:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0074 [2026-10-21 12:58:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6631 [2026-10-21 12:58:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4780 [2026-10-21 12:58:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6586 [2026-10-21 12:58:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4025 [2026-10-21 12:58:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5781 [2026-10-21 12:58:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.1515 [2026-10-21 12:58:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8267 [2026-10-21 12:58:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0384 [2026-10-21 12:59:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6100 [2026-10-21 12:59:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0086 [2026-10-21 12:59:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3125 [2026-10-21 12:59:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9579 [2026-10-21 12:59:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0598 [2026-10-21 12:59:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2167 [2026-10-21 12:59:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9739 [2026-10-21 12:59:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0660 [2026-10-21 12:59:44][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,632 | MFU: 1.8% [2026-10-21 12:59:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3082 [2026-10-21 12:59:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0601 [2026-10-21 12:59:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0108 [2026-10-21 12:59:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1785 [2026-10-21 12:59:49][INFO][training.pretrain_engine] [Ph2|Step 155] loss=5.2822 | ppl=196.80 | aux=0.02506 | lr=2.99e-04 | gnorm=0.629 | tok/s=29,377 | MFU=7.1% | step=142774ms | tokens=0.965B | mem=4.2/32.7GB | cost=$12.71/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 01:42 (132.7h) | Phase3: SKIPPED | SFT: Oct 27 06:26 (137.4h) | DPO (Final): Oct 27 07:00 (138.0h) [2026-10-21 12:59:49][INFO][training.pretrain_engine] Starting global step 156 (accumulating 32 micro-batches)... [2026-10-21 12:59:49][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 13:00:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1379 [2026-10-21 13:00:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1984 [2026-10-21 13:00:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9865 [2026-10-21 13:00:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6133 [2026-10-21 13:00:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4983 [2026-10-21 13:00:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5612 [2026-10-21 13:00:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4790 [2026-10-21 13:00:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6191 [2026-10-21 13:00:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8134 [2026-10-21 13:00:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2976 [2026-10-21 13:00:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2697 [2026-10-21 13:00:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7902 [2026-10-21 13:01:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2742 [2026-10-21 13:01:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4761 [2026-10-21 13:01:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2625 [2026-10-21 13:01:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0971 [2026-10-21 13:01:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1757 [2026-10-21 13:01:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1335 [2026-10-21 13:01:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0481 [2026-10-21 13:01:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4525 [2026-10-21 13:01:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0140 [2026-10-21 13:01:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8349 [2026-10-21 13:01:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2018 [2026-10-21 13:01:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3425 [2026-10-21 13:01:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2848 [2026-10-21 13:01:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3077 [2026-10-21 13:01:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3301 [2026-10-21 13:01:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2438 [2026-10-21 13:02:07][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,615 | MFU: 1.8% [2026-10-21 13:02:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2433 [2026-10-21 13:02:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4235 [2026-10-21 13:02:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1186 [2026-10-21 13:02:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4872 [2026-10-21 13:02:11][INFO][training.pretrain_engine] [Ph2|Step 156] loss=5.1813 | ppl=177.91 | aux=0.02506 | lr=2.99e-04 | gnorm=0.641 | tok/s=29,378 | MFU=7.1% | step=142771ms | tokens=0.966B | mem=4.2/32.7GB | cost=$12.80/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 01:44 (132.7h) | Phase3: SKIPPED | SFT: Oct 27 06:27 (137.4h) | DPO (Final): Oct 27 07:01 (138.0h) [2026-10-21 13:02:11][INFO][training.pretrain_engine] Starting global step 157 (accumulating 32 micro-batches)... [2026-10-21 13:02:11][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 13:02:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3129 [2026-10-21 13:02:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8617 [2026-10-21 13:02:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2972 [2026-10-21 13:02:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3157 [2026-10-21 13:02:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6543 [2026-10-21 13:02:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7232 [2026-10-21 13:02:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5660 [2026-10-21 13:02:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2283 [2026-10-21 13:03:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4288 [2026-10-21 13:03:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8239 [2026-10-21 13:03:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0407 [2026-10-21 13:03:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5690 [2026-10-21 13:03:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8686 [2026-10-21 13:03:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1590 [2026-10-21 13:03:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0836 [2026-10-21 13:03:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0458 [2026-10-21 13:03:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3086 [2026-10-21 13:03:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4058 [2026-10-21 13:03:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2300 [2026-10-21 13:03:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9223 [2026-10-21 13:03:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1628 [2026-10-21 13:03:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2074 [2026-10-21 13:03:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1064 [2026-10-21 13:03:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5833 [2026-10-21 13:04:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2633 [2026-10-21 13:04:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3462 [2026-10-21 13:04:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1858 [2026-10-21 13:04:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3650 [2026-10-21 13:04:29][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,624 | MFU: 1.8% [2026-10-21 13:04:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0056 [2026-10-21 13:04:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2534 [2026-10-21 13:04:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1677 [2026-10-21 13:04:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6620 [2026-10-21 13:04:33][INFO][training.pretrain_engine] [Ph2|Step 157] loss=5.2454 | ppl=189.69 | aux=0.02506 | lr=2.99e-04 | gnorm=0.668 | tok/s=29,380 | MFU=7.1% | step=142762ms | tokens=0.967B | mem=4.2/32.7GB | cost=$12.88/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 01:45 (132.7h) | Phase3: SKIPPED | SFT: Oct 27 06:29 (137.4h) | DPO (Final): Oct 27 07:03 (138.0h) [2026-10-21 13:04:33][INFO][training.pretrain_engine] Starting global step 158 (accumulating 32 micro-batches)... [2026-10-21 13:04:33][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 13:04:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9326 [2026-10-21 13:04:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5187 [2026-10-21 13:04:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3814 [2026-10-21 13:04:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8929 [2026-10-21 13:05:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2886 [2026-10-21 13:05:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1839 [2026-10-21 13:05:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4050 [2026-10-21 13:05:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2577 [2026-10-21 13:05:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5327 [2026-10-21 13:05:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4333 [2026-10-21 13:05:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7100 [2026-10-21 13:05:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5857 [2026-10-21 13:05:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1838 [2026-10-21 13:05:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1832 [2026-10-21 13:05:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5563 [2026-10-21 13:05:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3191 [2026-10-21 13:06:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6020 [2026-10-21 13:06:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0808 [2026-10-21 13:06:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4128 [2026-10-21 13:06:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4113 [2026-10-21 13:06:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1429 [2026-10-21 13:06:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2175 [2026-10-21 13:06:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6113 [2026-10-21 13:06:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1466 [2026-10-21 13:06:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3808 [2026-10-21 13:06:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3639 [2026-10-21 13:06:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6379 [2026-10-21 13:06:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3692 [2026-10-21 13:06:51][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,611 | MFU: 1.8% [2026-10-21 13:06:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8241 [2026-10-21 13:06:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3612 [2026-10-21 13:06:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0080 [2026-10-21 13:06:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0581 [2026-10-21 13:06:56][INFO][training.pretrain_engine] [Ph2|Step 158] loss=5.2066 | ppl=182.47 | aux=0.02507 | lr=2.99e-04 | gnorm=0.633 | tok/s=29,381 | MFU=7.1% | step=142754ms | tokens=0.968B | mem=4.2/32.7GB | cost=$12.96/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 01:46 (132.7h) | Phase3: SKIPPED | SFT: Oct 27 06:30 (137.4h) | DPO (Final): Oct 27 07:04 (138.0h) [2026-10-21 13:06:56][INFO][training.pretrain_engine] Starting global step 159 (accumulating 32 micro-batches)... [2026-10-21 13:06:56][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 13:07:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0902 [2026-10-21 13:07:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2790 [2026-10-21 13:07:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8707 [2026-10-21 13:07:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1600 [2026-10-21 13:07:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3974 [2026-10-21 13:07:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3236 [2026-10-21 13:07:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4957 [2026-10-21 13:07:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1222 [2026-10-21 13:07:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6355 [2026-10-21 13:07:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3961 [2026-10-21 13:07:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2970 [2026-10-21 13:07:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4746 [2026-10-21 13:08:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2957 [2026-10-21 13:08:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7356 [2026-10-21 13:08:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2737 [2026-10-21 13:08:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7865 [2026-10-21 13:08:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2859 [2026-10-21 13:08:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2884 [2026-10-21 13:08:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1534 [2026-10-21 13:08:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1567 [2026-10-21 13:08:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8131 [2026-10-21 13:08:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0656 [2026-10-21 13:08:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2151 [2026-10-21 13:08:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5499 [2026-10-21 13:08:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3957 [2026-10-21 13:09:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6091 [2026-10-21 13:09:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3886 [2026-10-21 13:09:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5282 [2026-10-21 13:09:13][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,613 | MFU: 1.8% [2026-10-21 13:09:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4160 [2026-10-21 13:09:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5749 [2026-10-21 13:09:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7038 [2026-10-21 13:09:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9861 [2026-10-21 13:09:18][INFO][training.pretrain_engine] [Ph2|Step 159] loss=5.2263 | ppl=186.10 | aux=0.02508 | lr=2.99e-04 | gnorm=0.699 | tok/s=29,382 | MFU=7.1% | step=142749ms | tokens=0.969B | mem=4.2/32.7GB | cost=$13.04/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 01:48 (132.6h) | Phase3: SKIPPED | SFT: Oct 27 06:31 (137.4h) | DPO (Final): Oct 27 07:05 (137.9h) [2026-10-21 13:09:18][INFO][training.pretrain_engine] Starting global step 160 (accumulating 32 micro-batches)... [2026-10-21 13:09:18][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 13:09:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5205 [2026-10-21 13:09:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9297 [2026-10-21 13:09:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9039 [2026-10-21 13:09:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3771 [2026-10-21 13:09:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1039 [2026-10-21 13:09:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4511 [2026-10-21 13:09:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2234 [2026-10-21 13:09:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3159 [2026-10-21 13:10:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4849 [2026-10-21 13:10:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6919 [2026-10-21 13:10:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9480 [2026-10-21 13:10:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6018 [2026-10-21 13:10:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2925 [2026-10-21 13:10:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4293 [2026-10-21 13:10:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0102 [2026-10-21 13:10:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9541 [2026-10-21 13:10:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5968 [2026-10-21 13:10:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0982 [2026-10-21 13:10:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6758 [2026-10-21 13:10:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7268 [2026-10-21 13:11:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2795 [2026-10-21 13:11:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8204 [2026-10-21 13:11:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0712 [2026-10-21 13:11:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3493 [2026-10-21 13:11:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5952 [2026-10-21 13:11:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1709 [2026-10-21 13:11:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1498 [2026-10-21 13:11:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2600 [2026-10-21 13:11:36][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,595 | MFU: 1.8% [2026-10-21 13:11:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1655 [2026-10-21 13:11:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2169 [2026-10-21 13:11:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2438 [2026-10-21 13:11:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4783 [2026-10-21 13:11:41][INFO][training.pretrain_engine] [Ph2|Step 160] loss=5.2471 | ppl=190.01 | aux=0.02507 | lr=2.99e-04 | gnorm=0.633 | tok/s=29,382 | MFU=7.1% | step=142749ms | tokens=0.970B | mem=4.2/32.7GB | cost=$13.12/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 01:50 (132.6h) | Phase3: SKIPPED | SFT: Oct 27 06:33 (137.4h) | DPO (Final): Oct 27 07:07 (137.9h) [2026-10-21 13:11:41][INFO][training.pretrain_engine] Starting global step 161 (accumulating 32 micro-batches)... [2026-10-21 13:11:41][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 13:11:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5578 [2026-10-21 13:11:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0002 [2026-10-21 13:11:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3538 [2026-10-21 13:11:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0802 [2026-10-21 13:12:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2333 [2026-10-21 13:12:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6276 [2026-10-21 13:12:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3855 [2026-10-21 13:12:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3493 [2026-10-21 13:12:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4513 [2026-10-21 13:12:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0090 [2026-10-21 13:12:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2474 [2026-10-21 13:12:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4356 [2026-10-21 13:12:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1873 [2026-10-21 13:12:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7879 [2026-10-21 13:12:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3858 [2026-10-21 13:12:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0634 [2026-10-21 13:13:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0780 [2026-10-21 13:13:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1758 [2026-10-21 13:13:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9218 [2026-10-21 13:13:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5003 [2026-10-21 13:13:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5567 [2026-10-21 13:13:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2677 [2026-10-21 13:13:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5516 [2026-10-21 13:13:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3047 [2026-10-21 13:13:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8616 [2026-10-21 13:13:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9978 [2026-10-21 13:13:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5517 [2026-10-21 13:13:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5807 [2026-10-21 13:13:58][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,615 | MFU: 1.8% [2026-10-21 13:14:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1695 [2026-10-21 13:14:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9396 [2026-10-21 13:14:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8284 [2026-10-21 13:14:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8233 [2026-10-21 13:14:04][INFO][training.pretrain_engine] [Ph2|Step 161] loss=5.1953 | ppl=180.41 | aux=0.02507 | lr=2.99e-04 | gnorm=0.672 | tok/s=29,381 | MFU=7.1% | step=142754ms | tokens=0.971B | mem=4.2/32.7GB | cost=$13.20/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 01:52 (132.6h) | Phase3: SKIPPED | SFT: Oct 27 06:35 (137.4h) | DPO (Final): Oct 27 07:09 (137.9h) [2026-10-21 13:14:04][INFO][training.pretrain_engine] Starting global step 162 (accumulating 32 micro-batches)... [2026-10-21 13:14:04][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 13:14:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4937 [2026-10-21 13:14:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9228 [2026-10-21 13:14:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2651 [2026-10-21 13:14:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6748 [2026-10-21 13:14:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3257 [2026-10-21 13:14:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2334 [2026-10-21 13:14:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4878 [2026-10-21 13:14:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1469 [2026-10-21 13:14:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4566 [2026-10-21 13:14:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2688 [2026-10-21 13:14:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1795 [2026-10-21 13:14:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2336 [2026-10-21 13:15:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4649 [2026-10-21 13:15:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5067 [2026-10-21 13:15:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5342 [2026-10-21 13:15:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8313 [2026-10-21 13:15:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0524 [2026-10-21 13:15:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7293 [2026-10-21 13:15:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6064 [2026-10-21 13:15:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5848 [2026-10-21 13:15:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3297 [2026-10-21 13:15:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7307 [2026-10-21 13:15:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0342 [2026-10-21 13:15:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2602 [2026-10-21 13:16:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7540 [2026-10-21 13:16:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1317 [2026-10-21 13:16:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3798 [2026-10-21 13:16:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3111 [2026-10-21 13:16:22][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,613 | MFU: 1.8% [2026-10-21 13:16:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3794 [2026-10-21 13:16:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5945 [2026-10-21 13:16:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2595 [2026-10-21 13:16:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5227 [2026-10-21 13:16:27][INFO][training.pretrain_engine] [Ph2|Step 162] loss=5.1898 | ppl=179.43 | aux=0.02507 | lr=2.99e-04 | gnorm=0.574 | tok/s=29,381 | MFU=7.1% | step=142754ms | tokens=0.972B | mem=4.2/32.7GB | cost=$13.28/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 01:53 (132.6h) | Phase3: SKIPPED | SFT: Oct 27 06:37 (137.4h) | DPO (Final): Oct 27 07:11 (137.9h) [2026-10-21 13:16:30][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step162 | Tokens: 0.972B | Spent: $13.29 | Remaining: $986.71 [2026-10-21 13:16:30][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step162 [2026-10-21 13:16:30][INFO][training.pretrain_engine] Starting global step 163 (accumulating 32 micro-batches)... [2026-10-21 13:16:30][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 13:16:30][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 13:16:30][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 13:16:30][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 13:16:30][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step162 to Phase-Technologies/forge-500m-pretrain (path: phase2_step162)... [2026-10-21 13:16:31][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 13:16:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2932 [2026-10-21 13:16:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0584 [2026-10-21 13:16:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0007 [2026-10-21 13:16:45][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 13:16:45][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step162 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 13:16:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2312 [2026-10-21 13:17:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4470 [2026-10-21 13:17:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0947 [2026-10-21 13:17:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5367 [2026-10-21 13:17:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5061 [2026-10-21 13:17:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7005 [2026-10-21 13:17:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2716 [2026-10-21 13:17:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4216 [2026-10-21 13:17:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2714 [2026-10-21 13:17:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4994 [2026-10-21 13:17:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3773 [2026-10-21 13:17:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2052 [2026-10-21 13:17:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3631 [2026-10-21 13:17:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1629 [2026-10-21 13:17:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1499 [2026-10-21 13:17:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5470 [2026-10-21 13:17:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4498 [2026-10-21 13:18:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2607 [2026-10-21 13:18:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0581 [2026-10-21 13:18:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1440 [2026-10-21 13:18:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2778 [2026-10-21 13:18:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0859 [2026-10-21 13:18:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3558 [2026-10-21 13:18:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5837 [2026-10-21 13:18:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5775 [2026-10-21 13:18:48][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,617 | MFU: 1.8% [2026-10-21 13:18:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0743 [2026-10-21 13:18:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5713 [2026-10-21 13:18:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1720 [2026-10-21 13:18:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0954 [2026-10-21 13:18:52][INFO][training.pretrain_engine] [Ph2|Step 163] loss=5.2920 | ppl=198.75 | aux=0.02506 | lr=2.99e-04 | gnorm=0.602 | tok/s=29,382 | MFU=7.1% | step=142750ms | tokens=0.973B | mem=4.2/32.7GB | cost=$13.37/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 01:55 (132.6h) | Phase3: SKIPPED | SFT: Oct 27 06:39 (137.3h) | DPO (Final): Oct 27 07:13 (137.9h) [2026-10-21 13:18:52][INFO][training.pretrain_engine] Starting global step 164 (accumulating 32 micro-batches)... [2026-10-21 13:18:52][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 13:19:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.9915 [2026-10-21 13:19:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0768 [2026-10-21 13:19:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2085 [2026-10-21 13:19:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1587 [2026-10-21 13:19:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2237 [2026-10-21 13:19:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2670 [2026-10-21 13:19:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6211 [2026-10-21 13:19:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0100 [2026-10-21 13:19:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3438 [2026-10-21 13:19:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5332 [2026-10-21 13:19:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3075 [2026-10-21 13:19:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4033 [2026-10-21 13:20:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9503 [2026-10-21 13:20:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0082 [2026-10-21 13:20:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0199 [2026-10-21 13:20:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7394 [2026-10-21 13:20:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3007 [2026-10-21 13:20:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3207 [2026-10-21 13:20:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0868 [2026-10-21 13:20:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5759 [2026-10-21 13:20:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4205 [2026-10-21 13:20:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3850 [2026-10-21 13:20:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3213 [2026-10-21 13:20:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3898 [2026-10-21 13:20:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6883 [2026-10-21 13:20:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5337 [2026-10-21 13:20:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7229 [2026-10-21 13:20:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7171 [2026-10-21 13:21:10][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,608 | MFU: 1.8% [2026-10-21 13:21:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2258 [2026-10-21 13:21:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0217 [2026-10-21 13:21:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1270 [2026-10-21 13:21:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9988 [2026-10-21 13:21:15][INFO][training.pretrain_engine] [Ph2|Step 164] loss=5.2429 | ppl=189.21 | aux=0.02508 | lr=2.99e-04 | gnorm=0.574 | tok/s=29,380 | MFU=7.1% | step=142759ms | tokens=0.974B | mem=4.2/32.7GB | cost=$13.45/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 01:57 (132.6h) | Phase3: SKIPPED | SFT: Oct 27 06:41 (137.3h) | DPO (Final): Oct 27 07:15 (137.9h) [2026-10-21 13:21:15][INFO][training.pretrain_engine] Starting global step 165 (accumulating 32 micro-batches)... [2026-10-21 13:21:15][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 13:21:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0309 [2026-10-21 13:21:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1757 [2026-10-21 13:21:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4797 [2026-10-21 13:21:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2357 [2026-10-21 13:21:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4655 [2026-10-21 13:21:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1522 [2026-10-21 13:21:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1064 [2026-10-21 13:21:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4898 [2026-10-21 13:22:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2014 [2026-10-21 13:22:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6773 [2026-10-21 13:22:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8837 [2026-10-21 13:22:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7991 [2026-10-21 13:22:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9128 [2026-10-21 13:22:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0168 [2026-10-21 13:22:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4007 [2026-10-21 13:22:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8782 [2026-10-21 13:22:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3922 [2026-10-21 13:22:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3617 [2026-10-21 13:22:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4248 [2026-10-21 13:22:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3857 [2026-10-21 13:23:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7133 [2026-10-21 13:23:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5001 [2026-10-21 13:23:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3802 [2026-10-21 13:23:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1851 [2026-10-21 13:23:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4392 [2026-10-21 13:23:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2041 [2026-10-21 13:23:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7402 [2026-10-21 13:23:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9530 [2026-10-21 13:23:32][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,636 | MFU: 1.8% [2026-10-21 13:23:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7705 [2026-10-21 13:23:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0889 [2026-10-21 13:23:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0559 [2026-10-21 13:23:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6800 [2026-10-21 13:23:38][INFO][training.pretrain_engine] [Ph2|Step 165] loss=5.2526 | ppl=191.06 | aux=0.02508 | lr=2.99e-04 | gnorm=0.512 | tok/s=29,380 | MFU=7.1% | step=142762ms | tokens=0.975B | mem=4.2/32.7GB | cost=$13.53/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 01:59 (132.6h) | Phase3: SKIPPED | SFT: Oct 27 06:43 (137.3h) | DPO (Final): Oct 27 07:17 (137.9h) [2026-10-21 13:23:38][INFO][training.pretrain_engine] Starting global step 166 (accumulating 32 micro-batches)... [2026-10-21 13:23:38][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 13:23:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9126 [2026-10-21 13:23:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3182 [2026-10-21 13:23:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4723 [2026-10-21 13:23:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4371 [2026-10-21 13:24:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2126 [2026-10-21 13:24:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3838 [2026-10-21 13:24:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4418 [2026-10-21 13:24:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7827 [2026-10-21 13:24:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1235 [2026-10-21 13:24:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9901 [2026-10-21 13:24:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3972 [2026-10-21 13:24:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0685 [2026-10-21 13:24:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2744 [2026-10-21 13:24:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5432 [2026-10-21 13:24:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9896 [2026-10-21 13:24:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4898 [2026-10-21 13:25:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4588 [2026-10-21 13:25:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3643 [2026-10-21 13:25:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3897 [2026-10-21 13:25:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5572 [2026-10-21 13:25:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9246 [2026-10-21 13:25:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3386 [2026-10-21 13:25:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3652 [2026-10-21 13:25:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6756 [2026-10-21 13:25:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4264 [2026-10-21 13:25:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3317 [2026-10-21 13:25:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9230 [2026-10-21 13:25:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4431 [2026-10-21 13:25:55][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,651 | MFU: 1.8% [2026-10-21 13:26:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4140 [2026-10-21 13:26:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4074 [2026-10-21 13:26:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2842 [2026-10-21 13:26:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4653 [2026-10-21 13:26:00][INFO][training.pretrain_engine] [Ph2|Step 166] loss=5.1827 | ppl=178.16 | aux=0.02506 | lr=2.99e-04 | gnorm=0.494 | tok/s=29,380 | MFU=7.1% | step=142759ms | tokens=0.976B | mem=4.2/32.7GB | cost=$13.61/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 02:01 (132.6h) | Phase3: SKIPPED | SFT: Oct 27 06:44 (137.3h) | DPO (Final): Oct 27 07:18 (137.9h) [2026-10-21 13:26:00][INFO][training.pretrain_engine] Starting global step 167 (accumulating 32 micro-batches)... [2026-10-21 13:26:00][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 13:26:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4046 [2026-10-21 13:26:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2049 [2026-10-21 13:26:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3907 [2026-10-21 13:26:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0822 [2026-10-21 13:26:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2859 [2026-10-21 13:26:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6081 [2026-10-21 13:26:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8337 [2026-10-21 13:26:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0038 [2026-10-21 13:26:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4650 [2026-10-21 13:26:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5022 [2026-10-21 13:26:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3852 [2026-10-21 13:26:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4307 [2026-10-21 13:27:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0486 [2026-10-21 13:27:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9769 [2026-10-21 13:27:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9111 [2026-10-21 13:27:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.1747 [2026-10-21 13:27:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=3.8366 [2026-10-21 13:27:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5164 [2026-10-21 13:27:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2619 [2026-10-21 13:27:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3434 [2026-10-21 13:27:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4158 [2026-10-21 13:27:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1919 [2026-10-21 13:27:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4067 [2026-10-21 13:27:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2384 [2026-10-21 13:28:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4244 [2026-10-21 13:28:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4097 [2026-10-21 13:28:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2186 [2026-10-21 13:28:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3166 [2026-10-21 13:28:17][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,623 | MFU: 1.8% [2026-10-21 13:28:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6019 [2026-10-21 13:28:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3698 [2026-10-21 13:28:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3123 [2026-10-21 13:28:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2054 [2026-10-21 13:28:22][INFO][training.pretrain_engine] [Ph2|Step 167] loss=5.2331 | ppl=187.38 | aux=0.02506 | lr=2.99e-04 | gnorm=0.479 | tok/s=29,381 | MFU=7.1% | step=142756ms | tokens=0.977B | mem=4.2/32.7GB | cost=$13.69/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 02:02 (132.6h) | Phase3: SKIPPED | SFT: Oct 27 06:46 (137.3h) | DPO (Final): Oct 27 07:20 (137.9h) [2026-10-21 13:28:22][INFO][training.pretrain_engine] Starting global step 168 (accumulating 32 micro-batches)... [2026-10-21 13:28:22][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 13:28:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2401 [2026-10-21 13:28:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9724 [2026-10-21 13:28:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1927 [2026-10-21 13:28:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1100 [2026-10-21 13:28:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0856 [2026-10-21 13:28:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4871 [2026-10-21 13:28:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7675 [2026-10-21 13:28:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1131 [2026-10-21 13:29:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1269 [2026-10-21 13:29:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7639 [2026-10-21 13:29:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8121 [2026-10-21 13:29:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9569 [2026-10-21 13:29:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2611 [2026-10-21 13:29:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3463 [2026-10-21 13:29:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4265 [2026-10-21 13:29:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4831 [2026-10-21 13:29:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5059 [2026-10-21 13:29:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3523 [2026-10-21 13:29:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2084 [2026-10-21 13:29:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1886 [2026-10-21 13:30:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6593 [2026-10-21 13:30:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0582 [2026-10-21 13:30:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9942 [2026-10-21 13:30:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6643 [2026-10-21 13:30:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8754 [2026-10-21 13:30:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6875 [2026-10-21 13:30:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0681 [2026-10-21 13:30:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6258 [2026-10-21 13:30:40][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,609 | MFU: 1.8% [2026-10-21 13:30:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9042 [2026-10-21 13:30:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6594 [2026-10-21 13:30:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0705 [2026-10-21 13:30:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5132 [2026-10-21 13:30:45][INFO][training.pretrain_engine] [Ph2|Step 168] loss=5.2568 | ppl=191.87 | aux=0.02506 | lr=2.99e-04 | gnorm=0.539 | tok/s=29,381 | MFU=7.1% | step=142756ms | tokens=0.978B | mem=4.2/32.7GB | cost=$13.78/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 02:04 (132.6h) | Phase3: SKIPPED | SFT: Oct 27 06:48 (137.3h) | DPO (Final): Oct 27 07:22 (137.9h) [2026-10-21 13:30:45][INFO][training.pretrain_engine] Starting global step 169 (accumulating 32 micro-batches)... [2026-10-21 13:30:45][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 13:31:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2616 [2026-10-21 13:31:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3738 [2026-10-21 13:31:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3570 [2026-10-21 13:31:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3490 [2026-10-21 13:31:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3360 [2026-10-21 13:31:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1966 [2026-10-21 13:31:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4801 [2026-10-21 13:31:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5532 [2026-10-21 13:31:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6287 [2026-10-21 13:31:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1638 [2026-10-21 13:31:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0332 [2026-10-21 13:31:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0352 [2026-10-21 13:31:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2971 [2026-10-21 13:31:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8161 [2026-10-21 13:31:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8406 [2026-10-21 13:31:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4815 [2026-10-21 13:32:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0338 [2026-10-21 13:32:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3705 [2026-10-21 13:32:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4922 [2026-10-21 13:32:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8802 [2026-10-21 13:32:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7043 [2026-10-21 13:32:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1575 [2026-10-21 13:32:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1662 [2026-10-21 13:32:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4338 [2026-10-21 13:32:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5318 [2026-10-21 13:32:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2211 [2026-10-21 13:32:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2716 [2026-10-21 13:32:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2366 [2026-10-21 13:33:02][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,611 | MFU: 1.8% [2026-10-21 13:33:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1869 [2026-10-21 13:33:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2632 [2026-10-21 13:33:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9511 [2026-10-21 13:33:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2407 [2026-10-21 13:33:07][INFO][training.pretrain_engine] [Ph2|Step 169] loss=5.2536 | ppl=191.25 | aux=0.02506 | lr=2.99e-04 | gnorm=0.684 | tok/s=29,381 | MFU=7.1% | step=142756ms | tokens=0.979B | mem=4.2/32.7GB | cost=$13.86/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 02:06 (132.6h) | Phase3: SKIPPED | SFT: Oct 27 06:50 (137.3h) | DPO (Final): Oct 27 07:24 (137.9h) [2026-10-21 13:33:07][INFO][training.pretrain_engine] Starting global step 170 (accumulating 32 micro-batches)... [2026-10-21 13:33:07][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 13:33:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5283 [2026-10-21 13:33:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8763 [2026-10-21 13:33:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4614 [2026-10-21 13:33:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5082 [2026-10-21 13:33:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8436 [2026-10-21 13:33:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9250 [2026-10-21 13:33:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1377 [2026-10-21 13:33:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0054 [2026-10-21 13:34:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1861 [2026-10-21 13:34:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9545 [2026-10-21 13:34:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6322 [2026-10-21 13:34:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7697 [2026-10-21 13:34:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4218 [2026-10-21 13:34:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4611 [2026-10-21 13:34:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6805 [2026-10-21 13:34:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0620 [2026-10-21 13:34:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4085 [2026-10-21 13:34:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9776 [2026-10-21 13:34:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2485 [2026-10-21 13:34:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3958 [2026-10-21 13:34:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5025 [2026-10-21 13:34:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1608 [2026-10-21 13:34:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3023 [2026-10-21 13:34:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6523 [2026-10-21 13:35:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4494 [2026-10-21 13:35:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3455 [2026-10-21 13:35:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4457 [2026-10-21 13:35:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4299 [2026-10-21 13:35:25][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,624 | MFU: 1.8% [2026-10-21 13:35:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2230 [2026-10-21 13:35:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5337 [2026-10-21 13:35:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5383 [2026-10-21 13:35:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5092 [2026-10-21 13:35:30][INFO][training.pretrain_engine] [Ph2|Step 170] loss=5.2754 | ppl=195.46 | aux=0.02506 | lr=2.99e-04 | gnorm=0.742 | tok/s=29,380 | MFU=7.1% | step=142761ms | tokens=0.980B | mem=4.2/32.7GB | cost=$13.94/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 02:08 (132.6h) | Phase3: SKIPPED | SFT: Oct 27 06:52 (137.3h) | DPO (Final): Oct 27 07:26 (137.8h) [2026-10-21 13:35:30][INFO][training.pretrain_engine] Starting global step 171 (accumulating 32 micro-batches)... [2026-10-21 13:35:30][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 13:35:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4687 [2026-10-21 13:35:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9265 [2026-10-21 13:35:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5858 [2026-10-21 13:35:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3174 [2026-10-21 13:36:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1021 [2026-10-21 13:36:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5865 [2026-10-21 13:36:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1992 [2026-10-21 13:36:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3514 [2026-10-21 13:36:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7115 [2026-10-21 13:36:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1285 [2026-10-21 13:36:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1221 [2026-10-21 13:36:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7957 [2026-10-21 13:36:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4002 [2026-10-21 13:36:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7843 [2026-10-21 13:36:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7897 [2026-10-21 13:36:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9928 [2026-10-21 13:36:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0598 [2026-10-21 13:36:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9020 [2026-10-21 13:36:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3207 [2026-10-21 13:36:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2517 [2026-10-21 13:37:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4552 [2026-10-21 13:37:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0807 [2026-10-21 13:37:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4349 [2026-10-21 13:37:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2238 [2026-10-21 13:37:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3701 [2026-10-21 13:37:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1529 [2026-10-21 13:37:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9121 [2026-10-21 13:37:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6013 [2026-10-21 13:37:47][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,622 | MFU: 1.8% [2026-10-21 13:37:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8744 [2026-10-21 13:37:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6999 [2026-10-21 13:37:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8294 [2026-10-21 13:37:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1149 [2026-10-21 13:37:52][INFO][training.pretrain_engine] [Ph2|Step 171] loss=5.1204 | ppl=167.40 | aux=0.02508 | lr=2.99e-04 | gnorm=0.684 | tok/s=29,380 | MFU=7.1% | step=142761ms | tokens=0.981B | mem=4.2/32.7GB | cost=$14.02/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 02:10 (132.5h) | Phase3: SKIPPED | SFT: Oct 27 06:53 (137.3h) | DPO (Final): Oct 27 07:27 (137.8h) [2026-10-21 13:37:55][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step171 | Tokens: 0.981B | Spent: $14.02 | Remaining: $985.98 [2026-10-21 13:37:55][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step171 [2026-10-21 13:37:55][INFO][training.pretrain_engine] Starting global step 172 (accumulating 32 micro-batches)... [2026-10-21 13:37:55][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 13:37:56][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 13:37:56][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 13:37:56][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 13:37:56][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step171 to Phase-Technologies/forge-500m-pretrain (path: phase2_step171)... [2026-10-21 13:37:56][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 13:38:06][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 13:38:06][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step171 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 13:38:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6081 [2026-10-21 13:38:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1043 [2026-10-21 13:38:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9504 [2026-10-21 13:38:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1117 [2026-10-21 13:38:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5679 [2026-10-21 13:38:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2413 [2026-10-21 13:38:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2423 [2026-10-21 13:38:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4284 [2026-10-21 13:38:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2530 [2026-10-21 13:38:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9506 [2026-10-21 13:38:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0172 [2026-10-21 13:38:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6337 [2026-10-21 13:39:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5009 [2026-10-21 13:39:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2417 [2026-10-21 13:39:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2628 [2026-10-21 13:39:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4974 [2026-10-21 13:39:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2913 [2026-10-21 13:39:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8009 [2026-10-21 13:39:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0554 [2026-10-21 13:39:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1669 [2026-10-21 13:39:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5155 [2026-10-21 13:39:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6018 [2026-10-21 13:39:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0362 [2026-10-21 13:39:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8916 [2026-10-21 13:39:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3358 [2026-10-21 13:39:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5291 [2026-10-21 13:39:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2040 [2026-10-21 13:40:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4363 [2026-10-21 13:40:13][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,618 | MFU: 1.8% [2026-10-21 13:40:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2785 [2026-10-21 13:40:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3425 [2026-10-21 13:40:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5273 [2026-10-21 13:40:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6942 [2026-10-21 13:40:18][INFO][training.pretrain_engine] [Ph2|Step 172] loss=5.1685 | ppl=175.64 | aux=0.02506 | lr=2.99e-04 | gnorm=0.582 | tok/s=29,381 | MFU=7.1% | step=142757ms | tokens=0.983B | mem=4.2/32.7GB | cost=$14.10/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 02:11 (132.5h) | Phase3: SKIPPED | SFT: Oct 27 06:55 (137.3h) | DPO (Final): Oct 27 07:29 (137.8h) [2026-10-21 13:40:18][INFO][training.pretrain_engine] Starting global step 173 (accumulating 32 micro-batches)... [2026-10-21 13:40:18][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 13:40:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9228 [2026-10-21 13:40:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1365 [2026-10-21 13:40:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0337 [2026-10-21 13:40:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0423 [2026-10-21 13:40:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8186 [2026-10-21 13:40:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6164 [2026-10-21 13:40:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5105 [2026-10-21 13:40:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7608 [2026-10-21 13:41:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3742 [2026-10-21 13:41:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1416 [2026-10-21 13:41:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9557 [2026-10-21 13:41:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4360 [2026-10-21 13:41:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7019 [2026-10-21 13:41:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1979 [2026-10-21 13:41:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2897 [2026-10-21 13:41:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5205 [2026-10-21 13:41:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7105 [2026-10-21 13:41:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.2309 [2026-10-21 13:41:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5290 [2026-10-21 13:41:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4419 [2026-10-21 13:42:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1476 [2026-10-21 13:42:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1321 [2026-10-21 13:42:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1186 [2026-10-21 13:42:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0059 [2026-10-21 13:42:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5218 [2026-10-21 13:42:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0917 [2026-10-21 13:42:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6028 [2026-10-21 13:42:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9947 [2026-10-21 13:42:35][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,608 | MFU: 1.8% [2026-10-21 13:42:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1624 [2026-10-21 13:42:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9354 [2026-10-21 13:42:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3995 [2026-10-21 13:42:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2702 [2026-10-21 13:42:41][INFO][training.pretrain_engine] [Ph2|Step 173] loss=5.2363 | ppl=187.97 | aux=0.02507 | lr=2.99e-04 | gnorm=0.605 | tok/s=29,378 | MFU=7.1% | step=142771ms | tokens=0.984B | mem=4.2/32.7GB | cost=$14.19/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 02:14 (132.5h) | Phase3: SKIPPED | SFT: Oct 27 06:58 (137.3h) | DPO (Final): Oct 27 07:32 (137.8h) [2026-10-21 13:42:41][INFO][training.pretrain_engine] Starting global step 174 (accumulating 32 micro-batches)... [2026-10-21 13:42:41][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 13:42:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1452 [2026-10-21 13:42:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9292 [2026-10-21 13:42:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6533 [2026-10-21 13:42:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2710 [2026-10-21 13:43:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0961 [2026-10-21 13:43:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9744 [2026-10-21 13:43:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1794 [2026-10-21 13:43:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9588 [2026-10-21 13:43:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6030 [2026-10-21 13:43:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4889 [2026-10-21 13:43:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3004 [2026-10-21 13:43:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1475 [2026-10-21 13:43:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0512 [2026-10-21 13:43:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5867 [2026-10-21 13:43:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2046 [2026-10-21 13:43:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1613 [2026-10-21 13:44:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3601 [2026-10-21 13:44:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1816 [2026-10-21 13:44:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4762 [2026-10-21 13:44:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9051 [2026-10-21 13:44:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0651 [2026-10-21 13:44:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9801 [2026-10-21 13:44:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4847 [2026-10-21 13:44:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2267 [2026-10-21 13:44:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1569 [2026-10-21 13:44:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.0757 [2026-10-21 13:44:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7950 [2026-10-21 13:44:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1467 [2026-10-21 13:44:58][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,619 | MFU: 1.8% [2026-10-21 13:45:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8623 [2026-10-21 13:45:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8241 [2026-10-21 13:45:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9133 [2026-10-21 13:45:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7533 [2026-10-21 13:45:04][INFO][training.pretrain_engine] [Ph2|Step 174] loss=5.1656 | ppl=175.14 | aux=0.02507 | lr=2.99e-04 | gnorm=0.793 | tok/s=29,378 | MFU=7.1% | step=142773ms | tokens=0.985B | mem=4.2/32.7GB | cost=$14.27/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 02:16 (132.5h) | Phase3: SKIPPED | SFT: Oct 27 07:00 (137.2h) | DPO (Final): Oct 27 07:34 (137.8h) [2026-10-21 13:45:04][INFO][training.pretrain_engine] Starting global step 175 (accumulating 32 micro-batches)... [2026-10-21 13:45:04][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 13:45:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3628 [2026-10-21 13:45:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5837 [2026-10-21 13:45:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5655 [2026-10-21 13:45:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8354 [2026-10-21 13:45:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5108 [2026-10-21 13:45:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1943 [2026-10-21 13:45:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3013 [2026-10-21 13:45:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4225 [2026-10-21 13:45:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0377 [2026-10-21 13:45:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9086 [2026-10-21 13:45:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0526 [2026-10-21 13:45:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8711 [2026-10-21 13:46:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5036 [2026-10-21 13:46:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2194 [2026-10-21 13:46:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1925 [2026-10-21 13:46:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0464 [2026-10-21 13:46:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1068 [2026-10-21 13:46:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8151 [2026-10-21 13:46:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7144 [2026-10-21 13:46:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2139 [2026-10-21 13:46:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2550 [2026-10-21 13:46:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5134 [2026-10-21 13:46:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7974 [2026-10-21 13:46:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6186 [2026-10-21 13:47:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4725 [2026-10-21 13:47:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5935 [2026-10-21 13:47:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3039 [2026-10-21 13:47:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5735 [2026-10-21 13:47:21][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,609 | MFU: 1.8% [2026-10-21 13:47:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3283 [2026-10-21 13:47:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4010 [2026-10-21 13:47:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5502 [2026-10-21 13:47:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2390 [2026-10-21 13:47:26][INFO][training.pretrain_engine] [Ph2|Step 175] loss=5.1680 | ppl=175.57 | aux=0.02507 | lr=2.99e-04 | gnorm=0.746 | tok/s=29,378 | MFU=7.1% | step=142769ms | tokens=0.986B | mem=4.2/32.7GB | cost=$14.35/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 02:17 (132.5h) | Phase3: SKIPPED | SFT: Oct 27 07:01 (137.2h) | DPO (Final): Oct 27 07:35 (137.8h) [2026-10-21 13:47:26][INFO][training.pretrain_engine] Starting global step 176 (accumulating 32 micro-batches)... [2026-10-21 13:47:26][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 13:47:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1823 [2026-10-21 13:47:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1728 [2026-10-21 13:47:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0823 [2026-10-21 13:47:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3007 [2026-10-21 13:48:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1699 [2026-10-21 13:48:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.3713 [2026-10-21 13:48:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6414 [2026-10-21 13:48:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3022 [2026-10-21 13:48:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0222 [2026-10-21 13:48:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9125 [2026-10-21 13:48:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7374 [2026-10-21 13:48:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5775 [2026-10-21 13:48:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2814 [2026-10-21 13:48:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2963 [2026-10-21 13:48:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3170 [2026-10-21 13:48:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0437 [2026-10-21 13:48:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9863 [2026-10-21 13:48:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2325 [2026-10-21 13:48:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9110 [2026-10-21 13:48:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2778 [2026-10-21 13:49:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2578 [2026-10-21 13:49:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4334 [2026-10-21 13:49:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0801 [2026-10-21 13:49:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5650 [2026-10-21 13:49:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7066 [2026-10-21 13:49:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1996 [2026-10-21 13:49:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8584 [2026-10-21 13:49:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1257 [2026-10-21 13:49:44][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,619 | MFU: 1.8% [2026-10-21 13:49:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6465 [2026-10-21 13:49:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9857 [2026-10-21 13:49:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7890 [2026-10-21 13:49:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9583 [2026-10-21 13:49:49][INFO][training.pretrain_engine] [Ph2|Step 176] loss=5.2922 | ppl=198.77 | aux=0.02507 | lr=2.99e-04 | gnorm=0.707 | tok/s=29,377 | MFU=7.1% | step=142775ms | tokens=0.987B | mem=4.2/32.7GB | cost=$14.43/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 02:20 (132.5h) | Phase3: SKIPPED | SFT: Oct 27 07:03 (137.2h) | DPO (Final): Oct 27 07:37 (137.8h) [2026-10-21 13:49:49][INFO][training.pretrain_engine] Starting global step 177 (accumulating 32 micro-batches)... [2026-10-21 13:49:49][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 13:50:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4098 [2026-10-21 13:50:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5734 [2026-10-21 13:50:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3358 [2026-10-21 13:50:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4928 [2026-10-21 13:50:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4449 [2026-10-21 13:50:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8074 [2026-10-21 13:50:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8648 [2026-10-21 13:50:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4508 [2026-10-21 13:50:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7030 [2026-10-21 13:50:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5979 [2026-10-21 13:50:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4768 [2026-10-21 13:50:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8224 [2026-10-21 13:51:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1042 [2026-10-21 13:51:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8322 [2026-10-21 13:51:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7086 [2026-10-21 13:51:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8123 [2026-10-21 13:51:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1306 [2026-10-21 13:51:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3626 [2026-10-21 13:51:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5089 [2026-10-21 13:51:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1187 [2026-10-21 13:51:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8103 [2026-10-21 13:51:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1247 [2026-10-21 13:51:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2937 [2026-10-21 13:51:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2578 [2026-10-21 13:51:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1568 [2026-10-21 13:51:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5217 [2026-10-21 13:51:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6337 [2026-10-21 13:51:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4247 [2026-10-21 13:52:07][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,614 | MFU: 1.8% [2026-10-21 13:52:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3020 [2026-10-21 13:52:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8319 [2026-10-21 13:52:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4162 [2026-10-21 13:52:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9436 [2026-10-21 13:52:12][INFO][training.pretrain_engine] [Ph2|Step 177] loss=5.2978 | ppl=199.89 | aux=0.02506 | lr=2.99e-04 | gnorm=0.777 | tok/s=29,376 | MFU=7.1% | step=142779ms | tokens=0.988B | mem=4.2/32.7GB | cost=$14.51/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 02:22 (132.5h) | Phase3: SKIPPED | SFT: Oct 27 07:05 (137.2h) | DPO (Final): Oct 27 07:39 (137.8h) [2026-10-21 13:52:12][INFO][training.pretrain_engine] Starting global step 178 (accumulating 32 micro-batches)... [2026-10-21 13:52:12][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 13:52:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5137 [2026-10-21 13:52:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5231 [2026-10-21 13:52:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3868 [2026-10-21 13:52:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9525 [2026-10-21 13:52:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6025 [2026-10-21 13:52:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3268 [2026-10-21 13:52:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0248 [2026-10-21 13:52:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9867 [2026-10-21 13:53:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0143 [2026-10-21 13:53:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1119 [2026-10-21 13:53:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7302 [2026-10-21 13:53:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1374 [2026-10-21 13:53:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9584 [2026-10-21 13:53:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7067 [2026-10-21 13:53:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.1477 [2026-10-21 13:53:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1668 [2026-10-21 13:53:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6899 [2026-10-21 13:53:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8276 [2026-10-21 13:53:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8549 [2026-10-21 13:53:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9232 [2026-10-21 13:53:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1971 [2026-10-21 13:53:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0116 [2026-10-21 13:53:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2121 [2026-10-21 13:53:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5236 [2026-10-21 13:54:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0391 [2026-10-21 13:54:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3014 [2026-10-21 13:54:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3000 [2026-10-21 13:54:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3380 [2026-10-21 13:54:29][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,642 | MFU: 1.8% [2026-10-21 13:54:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1964 [2026-10-21 13:54:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4587 [2026-10-21 13:54:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9498 [2026-10-21 13:54:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4460 [2026-10-21 13:54:35][INFO][training.pretrain_engine] [Ph2|Step 178] loss=5.2171 | ppl=184.41 | aux=0.02506 | lr=2.99e-04 | gnorm=0.652 | tok/s=29,376 | MFU=7.1% | step=142779ms | tokens=0.989B | mem=4.2/32.7GB | cost=$14.59/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 02:23 (132.5h) | Phase3: SKIPPED | SFT: Oct 27 07:07 (137.2h) | DPO (Final): Oct 27 07:41 (137.8h) [2026-10-21 13:54:35][INFO][training.pretrain_engine] Starting global step 179 (accumulating 32 micro-batches)... [2026-10-21 13:54:35][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 13:54:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6666 [2026-10-21 13:54:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7603 [2026-10-21 13:54:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3093 [2026-10-21 13:54:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2064 [2026-10-21 13:55:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9678 [2026-10-21 13:55:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5870 [2026-10-21 13:55:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1797 [2026-10-21 13:55:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9298 [2026-10-21 13:55:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3648 [2026-10-21 13:55:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8370 [2026-10-21 13:55:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0632 [2026-10-21 13:55:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3149 [2026-10-21 13:55:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3763 [2026-10-21 13:55:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9462 [2026-10-21 13:55:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8935 [2026-10-21 13:55:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8479 [2026-10-21 13:56:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1411 [2026-10-21 13:56:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5022 [2026-10-21 13:56:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3474 [2026-10-21 13:56:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5985 [2026-10-21 13:56:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0535 [2026-10-21 13:56:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0253 [2026-10-21 13:56:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5954 [2026-10-21 13:56:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8721 [2026-10-21 13:56:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4771 [2026-10-21 13:56:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3523 [2026-10-21 13:56:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5965 [2026-10-21 13:56:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6891 [2026-10-21 13:56:52][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,642 | MFU: 1.8% [2026-10-21 13:56:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1666 [2026-10-21 13:56:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8808 [2026-10-21 13:56:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3879 [2026-10-21 13:56:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6844 [2026-10-21 13:56:57][INFO][training.pretrain_engine] [Ph2|Step 179] loss=5.2760 | ppl=195.59 | aux=0.02506 | lr=2.99e-04 | gnorm=0.570 | tok/s=29,376 | MFU=7.1% | step=142780ms | tokens=0.990B | mem=4.2/32.7GB | cost=$14.68/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 02:25 (132.5h) | Phase3: SKIPPED | SFT: Oct 27 07:09 (137.2h) | DPO (Final): Oct 27 07:43 (137.8h) [2026-10-21 13:56:57][INFO][training.pretrain_engine] Starting global step 180 (accumulating 32 micro-batches)... [2026-10-21 13:56:57][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 13:57:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4015 [2026-10-21 13:57:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5201 [2026-10-21 13:57:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2938 [2026-10-21 13:57:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3401 [2026-10-21 13:57:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5990 [2026-10-21 13:57:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4719 [2026-10-21 13:57:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.0275 [2026-10-21 13:57:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6522 [2026-10-21 13:57:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9947 [2026-10-21 13:57:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5356 [2026-10-21 13:57:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2847 [2026-10-21 13:57:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2703 [2026-10-21 13:58:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7970 [2026-10-21 13:58:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9019 [2026-10-21 13:58:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5296 [2026-10-21 13:58:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1577 [2026-10-21 13:58:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4870 [2026-10-21 13:58:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1101 [2026-10-21 13:58:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6657 [2026-10-21 13:58:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7775 [2026-10-21 13:58:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1569 [2026-10-21 13:58:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0545 [2026-10-21 13:58:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1112 [2026-10-21 13:58:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8919 [2026-10-21 13:59:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7344 [2026-10-21 13:59:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8096 [2026-10-21 13:59:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2393 [2026-10-21 13:59:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6215 [2026-10-21 13:59:15][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,618 | MFU: 1.8% [2026-10-21 13:59:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5444 [2026-10-21 13:59:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5064 [2026-10-21 13:59:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4819 [2026-10-21 13:59:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5116 [2026-10-21 13:59:20][INFO][training.pretrain_engine] [Ph2|Step 180] loss=5.2201 | ppl=184.96 | aux=0.02509 | lr=2.99e-04 | gnorm=0.570 | tok/s=29,374 | MFU=7.1% | step=142788ms | tokens=0.991B | mem=4.2/32.7GB | cost=$14.76/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 02:27 (132.5h) | Phase3: SKIPPED | SFT: Oct 27 07:11 (137.2h) | DPO (Final): Oct 27 07:45 (137.8h) [2026-10-21 13:59:24][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step180 | Tokens: 0.991B | Spent: $14.76 | Remaining: $985.24 [2026-10-21 13:59:24][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step180 [2026-10-21 13:59:24][INFO][training.pretrain_engine] Starting global step 181 (accumulating 32 micro-batches)... [2026-10-21 13:59:24][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 13:59:24][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 13:59:24][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 13:59:24][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 13:59:24][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step180 to Phase-Technologies/forge-500m-pretrain (path: phase2_step180)... [2026-10-21 13:59:24][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 13:59:35][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 13:59:35][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step180 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 13:59:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6262 [2026-10-21 13:59:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1084 [2026-10-21 13:59:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2640 [2026-10-21 13:59:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6873 [2026-10-21 13:59:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1809 [2026-10-21 13:59:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0516 [2026-10-21 13:59:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8900 [2026-10-21 13:59:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2685 [2026-10-21 14:00:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6089 [2026-10-21 14:00:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4355 [2026-10-21 14:00:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3959 [2026-10-21 14:00:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2552 [2026-10-21 14:00:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5712 [2026-10-21 14:00:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8808 [2026-10-21 14:00:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3973 [2026-10-21 14:00:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7651 [2026-10-21 14:00:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6111 [2026-10-21 14:00:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2997 [2026-10-21 14:00:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6498 [2026-10-21 14:00:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1850 [2026-10-21 14:01:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4282 [2026-10-21 14:01:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1691 [2026-10-21 14:01:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4442 [2026-10-21 14:01:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4020 [2026-10-21 14:01:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9978 [2026-10-21 14:01:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2361 [2026-10-21 14:01:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1166 [2026-10-21 14:01:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4752 [2026-10-21 14:01:41][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,627 | MFU: 1.8% [2026-10-21 14:01:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4357 [2026-10-21 14:01:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2915 [2026-10-21 14:01:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6812 [2026-10-21 14:01:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3941 [2026-10-21 14:01:46][INFO][training.pretrain_engine] [Ph2|Step 181] loss=5.2519 | ppl=190.93 | aux=0.02506 | lr=2.99e-04 | gnorm=0.598 | tok/s=29,375 | MFU=7.1% | step=142784ms | tokens=0.992B | mem=4.2/32.7GB | cost=$14.84/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 02:29 (132.5h) | Phase3: SKIPPED | SFT: Oct 27 07:13 (137.2h) | DPO (Final): Oct 27 07:47 (137.8h) [2026-10-21 14:01:46][INFO][training.pretrain_engine] Starting global step 182 (accumulating 32 micro-batches)... [2026-10-21 14:01:46][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 14:02:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5500 [2026-10-21 14:02:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7916 [2026-10-21 14:02:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1539 [2026-10-21 14:02:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1528 [2026-10-21 14:02:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8552 [2026-10-21 14:02:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2292 [2026-10-21 14:02:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7435 [2026-10-21 14:02:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3785 [2026-10-21 14:02:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1165 [2026-10-21 14:02:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7578 [2026-10-21 14:02:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8006 [2026-10-21 14:02:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5044 [2026-10-21 14:02:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0465 [2026-10-21 14:02:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3791 [2026-10-21 14:02:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7821 [2026-10-21 14:02:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6496 [2026-10-21 14:03:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1842 [2026-10-21 14:03:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3771 [2026-10-21 14:03:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3908 [2026-10-21 14:03:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0237 [2026-10-21 14:03:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1151 [2026-10-21 14:03:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0130 [2026-10-21 14:03:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7980 [2026-10-21 14:03:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2721 [2026-10-21 14:03:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2122 [2026-10-21 14:03:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9472 [2026-10-21 14:03:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9180 [2026-10-21 14:03:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2846 [2026-10-21 14:04:03][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,646 | MFU: 1.8% [2026-10-21 14:04:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0733 [2026-10-21 14:04:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2309 [2026-10-21 14:04:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7908 [2026-10-21 14:04:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3136 [2026-10-21 14:04:09][INFO][training.pretrain_engine] [Ph2|Step 182] loss=5.2585 | ppl=192.20 | aux=0.02508 | lr=2.99e-04 | gnorm=0.664 | tok/s=29,373 | MFU=7.0% | step=142795ms | tokens=0.993B | mem=4.2/32.7GB | cost=$14.92/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 02:31 (132.5h) | Phase3: SKIPPED | SFT: Oct 27 07:15 (137.2h) | DPO (Final): Oct 27 07:49 (137.8h) [2026-10-21 14:04:09][INFO][training.pretrain_engine] Starting global step 183 (accumulating 32 micro-batches)... [2026-10-21 14:04:09][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 14:04:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5266 [2026-10-21 14:04:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7919 [2026-10-21 14:04:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0730 [2026-10-21 14:04:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9487 [2026-10-21 14:04:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6551 [2026-10-21 14:04:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9085 [2026-10-21 14:04:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2760 [2026-10-21 14:04:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1529 [2026-10-21 14:05:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4501 [2026-10-21 14:05:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6502 [2026-10-21 14:05:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6236 [2026-10-21 14:05:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2111 [2026-10-21 14:05:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3056 [2026-10-21 14:05:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8503 [2026-10-21 14:05:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0872 [2026-10-21 14:05:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4254 [2026-10-21 14:05:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4963 [2026-10-21 14:05:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2048 [2026-10-21 14:05:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1554 [2026-10-21 14:05:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2307 [2026-10-21 14:05:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2022 [2026-10-21 14:05:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3937 [2026-10-21 14:05:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1754 [2026-10-21 14:05:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1240 [2026-10-21 14:06:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5189 [2026-10-21 14:06:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2689 [2026-10-21 14:06:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0821 [2026-10-21 14:06:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9574 [2026-10-21 14:06:26][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,623 | MFU: 1.8% [2026-10-21 14:06:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9947 [2026-10-21 14:06:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3025 [2026-10-21 14:06:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9468 [2026-10-21 14:06:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2722 [2026-10-21 14:06:32][INFO][training.pretrain_engine] [Ph2|Step 183] loss=5.4006 | ppl=221.54 | aux=0.02506 | lr=2.99e-04 | gnorm=0.664 | tok/s=29,372 | MFU=7.0% | step=142801ms | tokens=0.994B | mem=4.2/32.7GB | cost=$15.00/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 02:34 (132.5h) | Phase3: SKIPPED | SFT: Oct 27 07:17 (137.2h) | DPO (Final): Oct 27 07:51 (137.8h) [2026-10-21 14:06:32][INFO][training.pretrain_engine] Starting global step 184 (accumulating 32 micro-batches)... [2026-10-21 14:06:32][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 14:06:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4371 [2026-10-21 14:06:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5120 [2026-10-21 14:06:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6856 [2026-10-21 14:06:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6967 [2026-10-21 14:07:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2789 [2026-10-21 14:07:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4109 [2026-10-21 14:07:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0625 [2026-10-21 14:07:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9376 [2026-10-21 14:07:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8554 [2026-10-21 14:07:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5343 [2026-10-21 14:07:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3148 [2026-10-21 14:07:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5886 [2026-10-21 14:07:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5525 [2026-10-21 14:07:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4309 [2026-10-21 14:07:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8265 [2026-10-21 14:07:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6906 [2026-10-21 14:08:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2320 [2026-10-21 14:08:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3576 [2026-10-21 14:08:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0229 [2026-10-21 14:08:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9302 [2026-10-21 14:08:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5794 [2026-10-21 14:08:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8663 [2026-10-21 14:08:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8956 [2026-10-21 14:08:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4345 [2026-10-21 14:08:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0423 [2026-10-21 14:08:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1259 [2026-10-21 14:08:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9213 [2026-10-21 14:08:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3749 [2026-10-21 14:08:49][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,626 | MFU: 1.8% [2026-10-21 14:08:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3935 [2026-10-21 14:08:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8663 [2026-10-21 14:08:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1840 [2026-10-21 14:08:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4143 [2026-10-21 14:08:55][INFO][training.pretrain_engine] [Ph2|Step 184] loss=5.2928 | ppl=198.91 | aux=0.02506 | lr=2.99e-04 | gnorm=0.691 | tok/s=29,371 | MFU=7.0% | step=142805ms | tokens=0.995B | mem=4.2/32.7GB | cost=$15.09/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 02:36 (132.5h) | Phase3: SKIPPED | SFT: Oct 27 07:19 (137.2h) | DPO (Final): Oct 27 07:53 (137.7h) [2026-10-21 14:08:55][INFO][training.pretrain_engine] Starting global step 185 (accumulating 32 micro-batches)... [2026-10-21 14:08:55][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 14:09:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6232 [2026-10-21 14:09:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0438 [2026-10-21 14:09:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5454 [2026-10-21 14:09:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9493 [2026-10-21 14:09:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4948 [2026-10-21 14:09:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3394 [2026-10-21 14:09:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6528 [2026-10-21 14:09:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3031 [2026-10-21 14:09:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7128 [2026-10-21 14:09:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4821 [2026-10-21 14:09:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6634 [2026-10-21 14:09:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0372 [2026-10-21 14:10:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3201 [2026-10-21 14:10:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5111 [2026-10-21 14:10:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1078 [2026-10-21 14:10:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2821 [2026-10-21 14:10:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4091 [2026-10-21 14:10:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6474 [2026-10-21 14:10:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6414 [2026-10-21 14:10:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4230 [2026-10-21 14:10:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7065 [2026-10-21 14:10:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8692 [2026-10-21 14:10:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6706 [2026-10-21 14:10:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0916 [2026-10-21 14:10:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1513 [2026-10-21 14:11:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2963 [2026-10-21 14:11:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4539 [2026-10-21 14:11:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2315 [2026-10-21 14:11:13][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,594 | MFU: 1.8% [2026-10-21 14:11:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2959 [2026-10-21 14:11:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2071 [2026-10-21 14:11:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5078 [2026-10-21 14:11:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4753 [2026-10-21 14:11:17][INFO][training.pretrain_engine] [Ph2|Step 185] loss=5.2486 | ppl=190.31 | aux=0.02507 | lr=2.99e-04 | gnorm=0.727 | tok/s=29,372 | MFU=7.0% | step=142801ms | tokens=0.996B | mem=4.2/32.7GB | cost=$15.17/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 02:37 (132.4h) | Phase3: SKIPPED | SFT: Oct 27 07:21 (137.2h) | DPO (Final): Oct 27 07:55 (137.7h) [2026-10-21 14:11:17][INFO][training.pretrain_engine] Starting global step 186 (accumulating 32 micro-batches)... [2026-10-21 14:11:17][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 14:11:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2886 [2026-10-21 14:11:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2300 [2026-10-21 14:11:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4286 [2026-10-21 14:11:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2464 [2026-10-21 14:11:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2145 [2026-10-21 14:11:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8371 [2026-10-21 14:11:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0265 [2026-10-21 14:11:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1093 [2026-10-21 14:12:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3194 [2026-10-21 14:12:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5750 [2026-10-21 14:12:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6296 [2026-10-21 14:12:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8839 [2026-10-21 14:12:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1343 [2026-10-21 14:12:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0997 [2026-10-21 14:12:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4636 [2026-10-21 14:12:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1173 [2026-10-21 14:12:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2875 [2026-10-21 14:12:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3148 [2026-10-21 14:12:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7403 [2026-10-21 14:12:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4769 [2026-10-21 14:13:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1289 [2026-10-21 14:13:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2169 [2026-10-21 14:13:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3520 [2026-10-21 14:13:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3240 [2026-10-21 14:13:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3801 [2026-10-21 14:13:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7286 [2026-10-21 14:13:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4873 [2026-10-21 14:13:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0396 [2026-10-21 14:13:35][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,627 | MFU: 1.8% [2026-10-21 14:13:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2801 [2026-10-21 14:13:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1545 [2026-10-21 14:13:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1572 [2026-10-21 14:13:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0959 [2026-10-21 14:13:40][INFO][training.pretrain_engine] [Ph2|Step 186] loss=5.2080 | ppl=182.73 | aux=0.02506 | lr=2.99e-04 | gnorm=0.770 | tok/s=29,373 | MFU=7.0% | step=142797ms | tokens=0.997B | mem=4.2/32.7GB | cost=$15.25/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 02:39 (132.4h) | Phase3: SKIPPED | SFT: Oct 27 07:22 (137.2h) | DPO (Final): Oct 27 07:56 (137.7h) [2026-10-21 14:13:40][INFO][training.pretrain_engine] Starting global step 187 (accumulating 32 micro-batches)... [2026-10-21 14:13:40][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 14:13:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4502 [2026-10-21 14:13:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1385 [2026-10-21 14:13:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5308 [2026-10-21 14:13:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2211 [2026-10-21 14:14:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1207 [2026-10-21 14:14:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0072 [2026-10-21 14:14:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5097 [2026-10-21 14:14:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0447 [2026-10-21 14:14:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8652 [2026-10-21 14:14:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3338 [2026-10-21 14:14:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1216 [2026-10-21 14:14:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1002 [2026-10-21 14:14:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0591 [2026-10-21 14:14:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4379 [2026-10-21 14:14:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3903 [2026-10-21 14:14:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5784 [2026-10-21 14:15:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1800 [2026-10-21 14:15:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0588 [2026-10-21 14:15:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9697 [2026-10-21 14:15:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5102 [2026-10-21 14:15:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8946 [2026-10-21 14:15:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3795 [2026-10-21 14:15:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3446 [2026-10-21 14:15:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3485 [2026-10-21 14:15:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1279 [2026-10-21 14:15:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7325 [2026-10-21 14:15:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8523 [2026-10-21 14:15:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8794 [2026-10-21 14:15:58][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,619 | MFU: 1.8% [2026-10-21 14:16:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2294 [2026-10-21 14:16:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2851 [2026-10-21 14:16:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5070 [2026-10-21 14:16:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0536 [2026-10-21 14:16:03][INFO][training.pretrain_engine] [Ph2|Step 187] loss=5.2595 | ppl=192.38 | aux=0.02508 | lr=2.99e-04 | gnorm=0.746 | tok/s=29,373 | MFU=7.0% | step=142793ms | tokens=0.998B | mem=4.2/32.7GB | cost=$15.33/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 02:40 (132.4h) | Phase3: SKIPPED | SFT: Oct 27 07:24 (137.1h) | DPO (Final): Oct 27 07:58 (137.7h) [2026-10-21 14:16:03][INFO][training.pretrain_engine] Starting global step 188 (accumulating 32 micro-batches)... [2026-10-21 14:16:03][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 14:16:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4639 [2026-10-21 14:16:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1912 [2026-10-21 14:16:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5211 [2026-10-21 14:16:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5853 [2026-10-21 14:16:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4217 [2026-10-21 14:16:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7030 [2026-10-21 14:16:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7452 [2026-10-21 14:16:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7549 [2026-10-21 14:16:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1699 [2026-10-21 14:16:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4169 [2026-10-21 14:16:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7919 [2026-10-21 14:16:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0897 [2026-10-21 14:17:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7071 [2026-10-21 14:17:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0978 [2026-10-21 14:17:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0127 [2026-10-21 14:17:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9536 [2026-10-21 14:17:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4210 [2026-10-21 14:17:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2201 [2026-10-21 14:17:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1849 [2026-10-21 14:17:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1104 [2026-10-21 14:17:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4525 [2026-10-21 14:17:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5732 [2026-10-21 14:17:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6374 [2026-10-21 14:17:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8144 [2026-10-21 14:18:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2724 [2026-10-21 14:18:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9905 [2026-10-21 14:18:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0636 [2026-10-21 14:18:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5096 [2026-10-21 14:18:21][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,628 | MFU: 1.8% [2026-10-21 14:18:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3877 [2026-10-21 14:18:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4152 [2026-10-21 14:18:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6457 [2026-10-21 14:18:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6550 [2026-10-21 14:18:26][INFO][training.pretrain_engine] [Ph2|Step 188] loss=5.2302 | ppl=186.83 | aux=0.02507 | lr=2.99e-04 | gnorm=0.625 | tok/s=29,374 | MFU=7.1% | step=142788ms | tokens=0.999B | mem=4.2/32.7GB | cost=$15.41/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 02:42 (132.4h) | Phase3: SKIPPED | SFT: Oct 27 07:25 (137.1h) | DPO (Final): Oct 27 08:00 (137.7h) [2026-10-21 14:18:26][INFO][training.pretrain_engine] Starting global step 189 (accumulating 32 micro-batches)... [2026-10-21 14:18:26][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 14:18:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6975 [2026-10-21 14:18:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8184 [2026-10-21 14:18:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9522 [2026-10-21 14:18:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8231 [2026-10-21 14:19:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5389 [2026-10-21 14:19:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3793 [2026-10-21 14:19:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5588 [2026-10-21 14:19:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7618 [2026-10-21 14:19:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5018 [2026-10-21 14:19:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4688 [2026-10-21 14:19:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5033 [2026-10-21 14:19:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3113 [2026-10-21 14:19:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3154 [2026-10-21 14:19:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7828 [2026-10-21 14:19:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1425 [2026-10-21 14:19:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7652 [2026-10-21 14:19:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9401 [2026-10-21 14:19:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5435 [2026-10-21 14:19:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3774 [2026-10-21 14:19:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3671 [2026-10-21 14:20:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9896 [2026-10-21 14:20:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1754 [2026-10-21 14:20:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1261 [2026-10-21 14:20:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8601 [2026-10-21 14:20:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3642 [2026-10-21 14:20:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4568 [2026-10-21 14:20:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3024 [2026-10-21 14:20:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5084 [2026-10-21 14:20:44][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,603 | MFU: 1.8% [2026-10-21 14:20:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8111 [2026-10-21 14:20:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7301 [2026-10-21 14:20:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0790 [2026-10-21 14:20:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9380 [2026-10-21 14:20:50][INFO][training.pretrain_engine] [Ph2|Step 189] loss=5.2414 | ppl=188.94 | aux=0.02505 | lr=2.99e-04 | gnorm=0.652 | tok/s=29,374 | MFU=7.1% | step=142788ms | tokens=1.000B | mem=4.2/32.7GB | cost=$15.50/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 02:44 (132.4h) | Phase3: SKIPPED | SFT: Oct 27 07:27 (137.1h) | DPO (Final): Oct 27 08:01 (137.7h) [2026-10-21 14:20:53][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step189 | Tokens: 1.000B | Spent: $15.50 | Remaining: $984.50 [2026-10-21 14:20:53][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step189 [2026-10-21 14:20:53][INFO][training.pretrain_engine] Starting global step 190 (accumulating 32 micro-batches)... [2026-10-21 14:20:53][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 14:20:53][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 14:20:53][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 14:20:53][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 14:20:53][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step189 to Phase-Technologies/forge-500m-pretrain (path: phase2_step189)... [2026-10-21 14:20:53][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 14:21:06][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 14:21:06][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step189 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 14:21:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4578 [2026-10-21 14:21:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4674 [2026-10-21 14:21:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0626 [2026-10-21 14:21:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9599 [2026-10-21 14:21:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4587 [2026-10-21 14:21:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8570 [2026-10-21 14:21:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7811 [2026-10-21 14:21:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1924 [2026-10-21 14:21:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5469 [2026-10-21 14:21:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7830 [2026-10-21 14:21:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8549 [2026-10-21 14:21:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4287 [2026-10-21 14:22:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4830 [2026-10-21 14:22:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3049 [2026-10-21 14:22:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4843 [2026-10-21 14:22:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4373 [2026-10-21 14:22:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9262 [2026-10-21 14:22:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3772 [2026-10-21 14:22:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8935 [2026-10-21 14:22:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6976 [2026-10-21 14:22:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9461 [2026-10-21 14:22:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9105 [2026-10-21 14:22:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5535 [2026-10-21 14:22:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8625 [2026-10-21 14:22:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1654 [2026-10-21 14:22:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4286 [2026-10-21 14:22:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2685 [2026-10-21 14:22:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4007 [2026-10-21 14:23:11][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,609 | MFU: 1.8% [2026-10-21 14:23:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0039 [2026-10-21 14:23:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8391 [2026-10-21 14:23:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9235 [2026-10-21 14:23:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3817 [2026-10-21 14:23:15][INFO][training.pretrain_engine] [Ph2|Step 190] loss=5.2302 | ppl=186.82 | aux=0.02507 | lr=2.99e-04 | gnorm=0.711 | tok/s=29,377 | MFU=7.1% | step=142776ms | tokens=1.001B | mem=4.2/32.7GB | cost=$15.58/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 02:45 (132.4h) | Phase3: SKIPPED | SFT: Oct 27 07:28 (137.1h) | DPO (Final): Oct 27 08:02 (137.7h) [2026-10-21 14:23:15][INFO][training.pretrain_engine] Starting global step 191 (accumulating 32 micro-batches)... [2026-10-21 14:23:15][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 14:23:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9923 [2026-10-21 14:23:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9985 [2026-10-21 14:23:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1188 [2026-10-21 14:23:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1805 [2026-10-21 14:23:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4531 [2026-10-21 14:23:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1943 [2026-10-21 14:23:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2607 [2026-10-21 14:23:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3937 [2026-10-21 14:24:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0016 [2026-10-21 14:24:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0801 [2026-10-21 14:24:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3941 [2026-10-21 14:24:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9862 [2026-10-21 14:24:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0194 [2026-10-21 14:24:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3297 [2026-10-21 14:24:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5454 [2026-10-21 14:24:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2504 [2026-10-21 14:24:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1092 [2026-10-21 14:24:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9774 [2026-10-21 14:24:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9870 [2026-10-21 14:24:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9462 [2026-10-21 14:25:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0264 [2026-10-21 14:25:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2591 [2026-10-21 14:25:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0071 [2026-10-21 14:25:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5225 [2026-10-21 14:25:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8118 [2026-10-21 14:25:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9419 [2026-10-21 14:25:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0987 [2026-10-21 14:25:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=3.8870 [2026-10-21 14:25:33][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,613 | MFU: 1.8% [2026-10-21 14:25:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7796 [2026-10-21 14:25:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0591 [2026-10-21 14:25:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1315 [2026-10-21 14:25:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0633 [2026-10-21 14:25:39][INFO][training.pretrain_engine] [Ph2|Step 191] loss=5.1995 | ppl=181.18 | aux=0.02507 | lr=2.99e-04 | gnorm=0.613 | tok/s=29,375 | MFU=7.1% | step=142786ms | tokens=1.002B | mem=4.2/32.7GB | cost=$15.66/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 02:47 (132.4h) | Phase3: SKIPPED | SFT: Oct 27 07:31 (137.1h) | DPO (Final): Oct 27 08:05 (137.7h) [2026-10-21 14:25:39][INFO][training.pretrain_engine] Starting global step 192 (accumulating 32 micro-batches)... [2026-10-21 14:25:39][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 14:25:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0833 [2026-10-21 14:25:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3305 [2026-10-21 14:25:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8087 [2026-10-21 14:25:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5790 [2026-10-21 14:26:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8694 [2026-10-21 14:26:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8641 [2026-10-21 14:26:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1613 [2026-10-21 14:26:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1584 [2026-10-21 14:26:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9030 [2026-10-21 14:26:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0401 [2026-10-21 14:26:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0738 [2026-10-21 14:26:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1877 [2026-10-21 14:26:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3840 [2026-10-21 14:26:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4480 [2026-10-21 14:26:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8482 [2026-10-21 14:26:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8704 [2026-10-21 14:27:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1124 [2026-10-21 14:27:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5028 [2026-10-21 14:27:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2256 [2026-10-21 14:27:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1247 [2026-10-21 14:27:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8656 [2026-10-21 14:27:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6211 [2026-10-21 14:27:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6988 [2026-10-21 14:27:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5131 [2026-10-21 14:27:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6770 [2026-10-21 14:27:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2798 [2026-10-21 14:27:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8218 [2026-10-21 14:27:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6784 [2026-10-21 14:27:56][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,611 | MFU: 1.8% [2026-10-21 14:28:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8737 [2026-10-21 14:28:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3238 [2026-10-21 14:28:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9009 [2026-10-21 14:28:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9091 [2026-10-21 14:28:02][INFO][training.pretrain_engine] [Ph2|Step 192] loss=5.2136 | ppl=183.76 | aux=0.02506 | lr=2.99e-04 | gnorm=0.508 | tok/s=29,376 | MFU=7.1% | step=142780ms | tokens=1.003B | mem=4.2/32.7GB | cost=$15.74/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 02:49 (132.4h) | Phase3: SKIPPED | SFT: Oct 27 07:32 (137.1h) | DPO (Final): Oct 27 08:06 (137.6h) [2026-10-21 14:28:02][INFO][training.pretrain_engine] Starting global step 193 (accumulating 32 micro-batches)... [2026-10-21 14:28:02][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 14:28:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6872 [2026-10-21 14:28:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5536 [2026-10-21 14:28:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5348 [2026-10-21 14:28:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4341 [2026-10-21 14:28:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9711 [2026-10-21 14:28:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6014 [2026-10-21 14:28:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7602 [2026-10-21 14:28:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4917 [2026-10-21 14:28:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0171 [2026-10-21 14:28:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3510 [2026-10-21 14:28:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0310 [2026-10-21 14:28:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8472 [2026-10-21 14:29:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0353 [2026-10-21 14:29:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2378 [2026-10-21 14:29:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2378 [2026-10-21 14:29:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4438 [2026-10-21 14:29:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2409 [2026-10-21 14:29:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7840 [2026-10-21 14:29:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3145 [2026-10-21 14:29:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0971 [2026-10-21 14:29:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7739 [2026-10-21 14:29:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5895 [2026-10-21 14:29:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1509 [2026-10-21 14:29:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3002 [2026-10-21 14:30:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2524 [2026-10-21 14:30:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9061 [2026-10-21 14:30:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8549 [2026-10-21 14:30:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3362 [2026-10-21 14:30:19][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,622 | MFU: 1.8% [2026-10-21 14:30:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8502 [2026-10-21 14:30:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9051 [2026-10-21 14:30:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0934 [2026-10-21 14:30:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6870 [2026-10-21 14:30:24][INFO][training.pretrain_engine] [Ph2|Step 193] loss=5.1793 | ppl=177.55 | aux=0.02506 | lr=2.99e-04 | gnorm=0.555 | tok/s=29,378 | MFU=7.1% | step=142772ms | tokens=1.005B | mem=4.2/32.7GB | cost=$15.82/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 02:50 (132.3h) | Phase3: SKIPPED | SFT: Oct 27 07:34 (137.1h) | DPO (Final): Oct 27 08:08 (137.6h) [2026-10-21 14:30:24][INFO][training.pretrain_engine] Starting global step 194 (accumulating 32 micro-batches)... [2026-10-21 14:30:24][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 14:30:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4635 [2026-10-21 14:30:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2398 [2026-10-21 14:30:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1830 [2026-10-21 14:30:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1810 [2026-10-21 14:31:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1828 [2026-10-21 14:31:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4187 [2026-10-21 14:31:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1718 [2026-10-21 14:31:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3504 [2026-10-21 14:31:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9152 [2026-10-21 14:31:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=3.8061 [2026-10-21 14:31:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6081 [2026-10-21 14:31:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7019 [2026-10-21 14:31:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1115 [2026-10-21 14:31:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9181 [2026-10-21 14:31:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3808 [2026-10-21 14:31:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1942 [2026-10-21 14:31:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2643 [2026-10-21 14:31:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2500 [2026-10-21 14:31:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5197 [2026-10-21 14:31:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3678 [2026-10-21 14:32:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6049 [2026-10-21 14:32:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0729 [2026-10-21 14:32:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8326 [2026-10-21 14:32:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5581 [2026-10-21 14:32:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8284 [2026-10-21 14:32:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3306 [2026-10-21 14:32:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6139 [2026-10-21 14:32:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2906 [2026-10-21 14:32:42][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,618 | MFU: 1.8% [2026-10-21 14:32:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2146 [2026-10-21 14:32:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4949 [2026-10-21 14:32:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2856 [2026-10-21 14:32:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0944 [2026-10-21 14:32:47][INFO][training.pretrain_engine] [Ph2|Step 194] loss=5.1331 | ppl=169.54 | aux=0.02507 | lr=2.99e-04 | gnorm=0.617 | tok/s=29,378 | MFU=7.1% | step=142770ms | tokens=1.006B | mem=4.2/32.7GB | cost=$15.91/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 02:52 (132.3h) | Phase3: SKIPPED | SFT: Oct 27 07:35 (137.0h) | DPO (Final): Oct 27 08:09 (137.6h) [2026-10-21 14:32:47][INFO][training.pretrain_engine] Starting global step 195 (accumulating 32 micro-batches)... [2026-10-21 14:32:47][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 14:33:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8525 [2026-10-21 14:33:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4848 [2026-10-21 14:33:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8871 [2026-10-21 14:33:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2133 [2026-10-21 14:33:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1568 [2026-10-21 14:33:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2366 [2026-10-21 14:33:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6019 [2026-10-21 14:33:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6611 [2026-10-21 14:33:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7789 [2026-10-21 14:33:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3721 [2026-10-21 14:33:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4587 [2026-10-21 14:33:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4027 [2026-10-21 14:33:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7936 [2026-10-21 14:33:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6701 [2026-10-21 14:33:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7262 [2026-10-21 14:33:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4945 [2026-10-21 14:34:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0133 [2026-10-21 14:34:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2308 [2026-10-21 14:34:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3477 [2026-10-21 14:34:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0742 [2026-10-21 14:34:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2612 [2026-10-21 14:34:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1188 [2026-10-21 14:34:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1649 [2026-10-21 14:34:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1088 [2026-10-21 14:34:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1674 [2026-10-21 14:34:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9642 [2026-10-21 14:34:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8914 [2026-10-21 14:34:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1714 [2026-10-21 14:35:05][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,631 | MFU: 1.8% [2026-10-21 14:35:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3315 [2026-10-21 14:35:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3890 [2026-10-21 14:35:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9664 [2026-10-21 14:35:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7681 [2026-10-21 14:35:10][INFO][training.pretrain_engine] [Ph2|Step 195] loss=5.2396 | ppl=188.60 | aux=0.02506 | lr=2.99e-04 | gnorm=0.555 | tok/s=29,378 | MFU=7.1% | step=142769ms | tokens=1.007B | mem=4.2/32.7GB | cost=$15.99/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 02:53 (132.3h) | Phase3: SKIPPED | SFT: Oct 27 07:37 (137.0h) | DPO (Final): Oct 27 08:11 (137.6h) [2026-10-21 14:35:10][INFO][training.pretrain_engine] Starting global step 196 (accumulating 32 micro-batches)... [2026-10-21 14:35:10][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 14:35:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3319 [2026-10-21 14:35:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1573 [2026-10-21 14:35:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3341 [2026-10-21 14:35:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6153 [2026-10-21 14:35:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8359 [2026-10-21 14:35:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7176 [2026-10-21 14:35:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8886 [2026-10-21 14:35:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0631 [2026-10-21 14:36:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1518 [2026-10-21 14:36:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2467 [2026-10-21 14:36:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9888 [2026-10-21 14:36:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9145 [2026-10-21 14:36:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2265 [2026-10-21 14:36:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7753 [2026-10-21 14:36:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9809 [2026-10-21 14:36:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4333 [2026-10-21 14:36:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8177 [2026-10-21 14:36:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1645 [2026-10-21 14:36:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.1585 [2026-10-21 14:36:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6190 [2026-10-21 14:36:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8057 [2026-10-21 14:36:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5886 [2026-10-21 14:36:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8747 [2026-10-21 14:36:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5285 [2026-10-21 14:37:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1241 [2026-10-21 14:37:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4825 [2026-10-21 14:37:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6123 [2026-10-21 14:37:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4143 [2026-10-21 14:37:28][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,629 | MFU: 1.8% [2026-10-21 14:37:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6335 [2026-10-21 14:37:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1710 [2026-10-21 14:37:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5343 [2026-10-21 14:37:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6709 [2026-10-21 14:37:33][INFO][training.pretrain_engine] [Ph2|Step 196] loss=5.2247 | ppl=185.80 | aux=0.02506 | lr=2.99e-04 | gnorm=0.578 | tok/s=29,380 | MFU=7.1% | step=142759ms | tokens=1.008B | mem=4.2/32.7GB | cost=$16.07/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 02:55 (132.3h) | Phase3: SKIPPED | SFT: Oct 27 07:38 (137.0h) | DPO (Final): Oct 27 08:12 (137.6h) [2026-10-21 14:37:33][INFO][training.pretrain_engine] Starting global step 197 (accumulating 32 micro-batches)... [2026-10-21 14:37:33][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 14:37:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3961 [2026-10-21 14:37:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2242 [2026-10-21 14:37:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9940 [2026-10-21 14:37:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8089 [2026-10-21 14:38:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8916 [2026-10-21 14:38:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1137 [2026-10-21 14:38:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3009 [2026-10-21 14:38:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1356 [2026-10-21 14:38:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2743 [2026-10-21 14:38:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2173 [2026-10-21 14:38:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0889 [2026-10-21 14:38:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2681 [2026-10-21 14:38:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6723 [2026-10-21 14:38:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3595 [2026-10-21 14:38:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2997 [2026-10-21 14:38:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5801 [2026-10-21 14:39:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3665 [2026-10-21 14:39:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4096 [2026-10-21 14:39:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1097 [2026-10-21 14:39:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8867 [2026-10-21 14:39:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2837 [2026-10-21 14:39:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8976 [2026-10-21 14:39:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5838 [2026-10-21 14:39:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.1747 [2026-10-21 14:39:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4429 [2026-10-21 14:39:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9627 [2026-10-21 14:39:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3395 [2026-10-21 14:39:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1938 [2026-10-21 14:39:51][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,612 | MFU: 1.8% [2026-10-21 14:39:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5192 [2026-10-21 14:39:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9732 [2026-10-21 14:39:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0576 [2026-10-21 14:39:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0361 [2026-10-21 14:39:56][INFO][training.pretrain_engine] [Ph2|Step 197] loss=5.2316 | ppl=187.08 | aux=0.02506 | lr=2.99e-04 | gnorm=0.555 | tok/s=29,382 | MFU=7.1% | step=142753ms | tokens=1.009B | mem=4.2/32.7GB | cost=$16.15/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 02:56 (132.3h) | Phase3: SKIPPED | SFT: Oct 27 07:40 (137.0h) | DPO (Final): Oct 27 08:14 (137.6h) [2026-10-21 14:39:56][INFO][training.pretrain_engine] Starting global step 198 (accumulating 32 micro-batches)... [2026-10-21 14:39:56][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 14:40:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4804 [2026-10-21 14:40:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9601 [2026-10-21 14:40:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3460 [2026-10-21 14:40:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3036 [2026-10-21 14:40:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6005 [2026-10-21 14:40:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0116 [2026-10-21 14:40:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1647 [2026-10-21 14:40:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9423 [2026-10-21 14:40:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9493 [2026-10-21 14:40:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9808 [2026-10-21 14:40:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2120 [2026-10-21 14:40:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2418 [2026-10-21 14:41:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4050 [2026-10-21 14:41:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4713 [2026-10-21 14:41:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4500 [2026-10-21 14:41:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5671 [2026-10-21 14:41:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3046 [2026-10-21 14:41:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1920 [2026-10-21 14:41:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1616 [2026-10-21 14:41:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3048 [2026-10-21 14:41:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9689 [2026-10-21 14:41:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5181 [2026-10-21 14:41:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6086 [2026-10-21 14:41:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6612 [2026-10-21 14:42:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7052 [2026-10-21 14:42:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4865 [2026-10-21 14:42:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7187 [2026-10-21 14:42:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3399 [2026-10-21 14:42:13][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,621 | MFU: 1.8% [2026-10-21 14:42:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3081 [2026-10-21 14:42:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3835 [2026-10-21 14:42:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4435 [2026-10-21 14:42:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3644 [2026-10-21 14:42:18][INFO][training.pretrain_engine] [Ph2|Step 198] loss=5.2010 | ppl=181.46 | aux=0.02506 | lr=2.99e-04 | gnorm=0.490 | tok/s=29,382 | MFU=7.1% | step=142749ms | tokens=1.010B | mem=4.2/32.7GB | cost=$16.23/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 02:58 (132.3h) | Phase3: SKIPPED | SFT: Oct 27 07:41 (137.0h) | DPO (Final): Oct 27 08:15 (137.6h) [2026-10-21 14:42:22][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step198 | Tokens: 1.010B | Spent: $16.23 | Remaining: $983.77 [2026-10-21 14:42:22][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step198 [2026-10-21 14:42:22][INFO][training.pretrain_engine] Starting global step 199 (accumulating 32 micro-batches)... [2026-10-21 14:42:22][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 14:42:22][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 14:42:22][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 14:42:22][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 14:42:22][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step198 to Phase-Technologies/forge-500m-pretrain (path: phase2_step198)... [2026-10-21 14:42:22][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 14:42:32][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 14:42:32][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step198 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 14:42:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1473 [2026-10-21 14:42:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2051 [2026-10-21 14:42:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6127 [2026-10-21 14:42:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7755 [2026-10-21 14:42:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3522 [2026-10-21 14:42:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1065 [2026-10-21 14:42:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1277 [2026-10-21 14:42:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8822 [2026-10-21 14:43:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0509 [2026-10-21 14:43:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1915 [2026-10-21 14:43:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4282 [2026-10-21 14:43:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1458 [2026-10-21 14:43:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3170 [2026-10-21 14:43:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5018 [2026-10-21 14:43:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2261 [2026-10-21 14:43:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1576 [2026-10-21 14:43:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3719 [2026-10-21 14:43:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3307 [2026-10-21 14:43:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5777 [2026-10-21 14:43:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1807 [2026-10-21 14:44:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6821 [2026-10-21 14:44:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3378 [2026-10-21 14:44:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1407 [2026-10-21 14:44:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4129 [2026-10-21 14:44:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2284 [2026-10-21 14:44:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2482 [2026-10-21 14:44:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7531 [2026-10-21 14:44:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4264 [2026-10-21 14:44:40][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,597 | MFU: 1.8% [2026-10-21 14:44:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0745 [2026-10-21 14:44:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1375 [2026-10-21 14:44:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5435 [2026-10-21 14:44:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5272 [2026-10-21 14:44:44][INFO][training.pretrain_engine] [Ph2|Step 199] loss=5.2733 | ppl=195.07 | aux=0.02506 | lr=2.99e-04 | gnorm=0.527 | tok/s=29,383 | MFU=7.1% | step=142746ms | tokens=1.011B | mem=4.2/32.7GB | cost=$16.32/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 02:59 (132.2h) | Phase3: SKIPPED | SFT: Oct 27 07:43 (137.0h) | DPO (Final): Oct 27 08:17 (137.5h) [2026-10-21 14:44:44][INFO][training.pretrain_engine] Starting global step 200 (accumulating 32 micro-batches)... [2026-10-21 14:44:44][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 14:45:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6882 [2026-10-21 14:45:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0221 [2026-10-21 14:45:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6997 [2026-10-21 14:45:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8608 [2026-10-21 14:45:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4456 [2026-10-21 14:45:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2068 [2026-10-21 14:45:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4088 [2026-10-21 14:45:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3801 [2026-10-21 14:45:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5623 [2026-10-21 14:45:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9048 [2026-10-21 14:45:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4077 [2026-10-21 14:45:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6952 [2026-10-21 14:45:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3269 [2026-10-21 14:45:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0380 [2026-10-21 14:45:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0467 [2026-10-21 14:45:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2207 [2026-10-21 14:46:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5731 [2026-10-21 14:46:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5871 [2026-10-21 14:46:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5794 [2026-10-21 14:46:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6639 [2026-10-21 14:46:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4633 [2026-10-21 14:46:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0662 [2026-10-21 14:46:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2693 [2026-10-21 14:46:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3125 [2026-10-21 14:46:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3691 [2026-10-21 14:46:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2392 [2026-10-21 14:46:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6283 [2026-10-21 14:46:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6836 [2026-10-21 14:47:02][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,603 | MFU: 1.8% [2026-10-21 14:47:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5550 [2026-10-21 14:47:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1536 [2026-10-21 14:47:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4226 [2026-10-21 14:47:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5110 [2026-10-21 14:47:07][INFO][training.pretrain_engine] [Ph2|Step 200] loss=5.2981 | ppl=199.97 | aux=0.02507 | lr=2.99e-04 | gnorm=0.629 | tok/s=29,381 | MFU=7.1% | step=142754ms | tokens=1.012B | mem=4.2/32.7GB | cost=$16.40/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 03:01 (132.2h) | Phase3: SKIPPED | SFT: Oct 27 07:45 (137.0h) | DPO (Final): Oct 27 08:19 (137.5h) [2026-10-21 14:47:07][INFO][training.pretrain_engine] Starting global step 201 (accumulating 32 micro-batches)... [2026-10-21 14:47:07][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 14:47:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2919 [2026-10-21 14:47:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9140 [2026-10-21 14:47:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3806 [2026-10-21 14:47:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0536 [2026-10-21 14:47:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0119 [2026-10-21 14:47:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9795 [2026-10-21 14:47:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8101 [2026-10-21 14:47:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0926 [2026-10-21 14:48:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2008 [2026-10-21 14:48:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9987 [2026-10-21 14:48:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8792 [2026-10-21 14:48:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9342 [2026-10-21 14:48:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7767 [2026-10-21 14:48:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0055 [2026-10-21 14:48:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3152 [2026-10-21 14:48:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1966 [2026-10-21 14:48:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8629 [2026-10-21 14:48:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8669 [2026-10-21 14:48:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4145 [2026-10-21 14:48:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2380 [2026-10-21 14:48:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2645 [2026-10-21 14:48:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5796 [2026-10-21 14:48:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6737 [2026-10-21 14:48:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5750 [2026-10-21 14:49:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8016 [2026-10-21 14:49:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0810 [2026-10-21 14:49:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4593 [2026-10-21 14:49:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9372 [2026-10-21 14:49:25][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,600 | MFU: 1.8% [2026-10-21 14:49:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1520 [2026-10-21 14:49:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3945 [2026-10-21 14:49:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7407 [2026-10-21 14:49:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9267 [2026-10-21 14:49:30][INFO][training.pretrain_engine] [Ph2|Step 201] loss=5.1905 | ppl=179.56 | aux=0.02506 | lr=2.99e-04 | gnorm=0.688 | tok/s=29,382 | MFU=7.1% | step=142753ms | tokens=1.013B | mem=4.2/32.7GB | cost=$16.48/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 03:03 (132.2h) | Phase3: SKIPPED | SFT: Oct 27 07:47 (137.0h) | DPO (Final): Oct 27 08:21 (137.5h) [2026-10-21 14:49:30][INFO][training.pretrain_engine] Starting global step 202 (accumulating 32 micro-batches)... [2026-10-21 14:49:30][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 14:49:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4297 [2026-10-21 14:49:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2518 [2026-10-21 14:49:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6673 [2026-10-21 14:49:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3033 [2026-10-21 14:50:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0629 [2026-10-21 14:50:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0992 [2026-10-21 14:50:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0578 [2026-10-21 14:50:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4354 [2026-10-21 14:50:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7266 [2026-10-21 14:50:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2615 [2026-10-21 14:50:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2315 [2026-10-21 14:50:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8658 [2026-10-21 14:50:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1035 [2026-10-21 14:50:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7996 [2026-10-21 14:50:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4376 [2026-10-21 14:50:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0050 [2026-10-21 14:50:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7700 [2026-10-21 14:50:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0415 [2026-10-21 14:50:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2594 [2026-10-21 14:50:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0006 [2026-10-21 14:51:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3940 [2026-10-21 14:51:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2232 [2026-10-21 14:51:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3966 [2026-10-21 14:51:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1593 [2026-10-21 14:51:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9195 [2026-10-21 14:51:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9930 [2026-10-21 14:51:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9031 [2026-10-21 14:51:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3850 [2026-10-21 14:51:48][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,597 | MFU: 1.8% [2026-10-21 14:51:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7361 [2026-10-21 14:51:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2921 [2026-10-21 14:51:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3194 [2026-10-21 14:51:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0623 [2026-10-21 14:51:53][INFO][training.pretrain_engine] [Ph2|Step 202] loss=5.1495 | ppl=172.35 | aux=0.02507 | lr=2.99e-04 | gnorm=0.875 | tok/s=29,382 | MFU=7.1% | step=142750ms | tokens=1.014B | mem=4.2/32.7GB | cost=$16.56/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 03:05 (132.2h) | Phase3: SKIPPED | SFT: Oct 27 07:48 (137.0h) | DPO (Final): Oct 27 08:22 (137.5h) [2026-10-21 14:51:53][INFO][training.pretrain_engine] Starting global step 203 (accumulating 32 micro-batches)... [2026-10-21 14:51:53][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 14:52:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5202 [2026-10-21 14:52:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2267 [2026-10-21 14:52:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3490 [2026-10-21 14:52:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4566 [2026-10-21 14:52:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2710 [2026-10-21 14:52:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5673 [2026-10-21 14:52:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6372 [2026-10-21 14:52:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3759 [2026-10-21 14:52:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3479 [2026-10-21 14:52:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3501 [2026-10-21 14:52:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3818 [2026-10-21 14:52:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9994 [2026-10-21 14:53:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4871 [2026-10-21 14:53:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6725 [2026-10-21 14:53:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1655 [2026-10-21 14:53:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3065 [2026-10-21 14:53:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9454 [2026-10-21 14:53:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8630 [2026-10-21 14:53:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0315 [2026-10-21 14:53:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1954 [2026-10-21 14:53:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9217 [2026-10-21 14:53:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8079 [2026-10-21 14:53:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5382 [2026-10-21 14:53:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4555 [2026-10-21 14:53:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4246 [2026-10-21 14:53:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3076 [2026-10-21 14:53:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4207 [2026-10-21 14:53:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4977 [2026-10-21 14:54:10][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,631 | MFU: 1.8% [2026-10-21 14:54:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6020 [2026-10-21 14:54:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5104 [2026-10-21 14:54:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1736 [2026-10-21 14:54:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5334 [2026-10-21 14:54:15][INFO][training.pretrain_engine] [Ph2|Step 203] loss=5.2319 | ppl=187.14 | aux=0.02507 | lr=2.99e-04 | gnorm=0.949 | tok/s=29,384 | MFU=7.1% | step=142743ms | tokens=1.015B | mem=4.2/32.7GB | cost=$16.64/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 03:06 (132.2h) | Phase3: SKIPPED | SFT: Oct 27 07:50 (136.9h) | DPO (Final): Oct 27 08:24 (137.5h) [2026-10-21 14:54:15][INFO][training.pretrain_engine] Starting global step 204 (accumulating 32 micro-batches)... [2026-10-21 14:54:15][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 14:54:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4400 [2026-10-21 14:54:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2958 [2026-10-21 14:54:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7846 [2026-10-21 14:54:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1656 [2026-10-21 14:54:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5329 [2026-10-21 14:54:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5466 [2026-10-21 14:54:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5447 [2026-10-21 14:54:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0935 [2026-10-21 14:55:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4855 [2026-10-21 14:55:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2990 [2026-10-21 14:55:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8154 [2026-10-21 14:55:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9195 [2026-10-21 14:55:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9623 [2026-10-21 14:55:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0863 [2026-10-21 14:55:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7581 [2026-10-21 14:55:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1780 [2026-10-21 14:55:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0179 [2026-10-21 14:55:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6037 [2026-10-21 14:55:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0357 [2026-10-21 14:55:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7284 [2026-10-21 14:56:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3991 [2026-10-21 14:56:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0138 [2026-10-21 14:56:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2962 [2026-10-21 14:56:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5974 [2026-10-21 14:56:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8536 [2026-10-21 14:56:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7596 [2026-10-21 14:56:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5275 [2026-10-21 14:56:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1556 [2026-10-21 14:56:33][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,633 | MFU: 1.8% [2026-10-21 14:56:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1813 [2026-10-21 14:56:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2874 [2026-10-21 14:56:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7639 [2026-10-21 14:56:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4467 [2026-10-21 14:56:38][INFO][training.pretrain_engine] [Ph2|Step 204] loss=5.1102 | ppl=165.71 | aux=0.02507 | lr=2.99e-04 | gnorm=0.961 | tok/s=29,387 | MFU=7.1% | step=142729ms | tokens=1.016B | mem=4.2/32.7GB | cost=$16.72/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 03:07 (132.2h) | Phase3: SKIPPED | SFT: Oct 27 07:51 (136.9h) | DPO (Final): Oct 27 08:25 (137.5h) [2026-10-21 14:56:38][INFO][training.pretrain_engine] Starting global step 205 (accumulating 32 micro-batches)... [2026-10-21 14:56:38][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 14:56:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3806 [2026-10-21 14:56:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4185 [2026-10-21 14:56:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8044 [2026-10-21 14:56:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1606 [2026-10-21 14:57:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0845 [2026-10-21 14:57:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0315 [2026-10-21 14:57:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9509 [2026-10-21 14:57:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3453 [2026-10-21 14:57:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1232 [2026-10-21 14:57:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0656 [2026-10-21 14:57:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8650 [2026-10-21 14:57:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1560 [2026-10-21 14:57:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6172 [2026-10-21 14:57:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6921 [2026-10-21 14:57:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7490 [2026-10-21 14:57:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8701 [2026-10-21 14:58:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4477 [2026-10-21 14:58:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7720 [2026-10-21 14:58:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7656 [2026-10-21 14:58:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4348 [2026-10-21 14:58:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8900 [2026-10-21 14:58:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2028 [2026-10-21 14:58:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6457 [2026-10-21 14:58:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6415 [2026-10-21 14:58:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0238 [2026-10-21 14:58:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8963 [2026-10-21 14:58:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2612 [2026-10-21 14:58:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1494 [2026-10-21 14:58:55][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,629 | MFU: 1.8% [2026-10-21 14:59:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9741 [2026-10-21 14:59:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0545 [2026-10-21 14:59:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6146 [2026-10-21 14:59:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9634 [2026-10-21 14:59:00][INFO][training.pretrain_engine] [Ph2|Step 205] loss=5.0541 | ppl=156.67 | aux=0.02507 | lr=2.99e-04 | gnorm=0.867 | tok/s=29,387 | MFU=7.1% | step=142725ms | tokens=1.017B | mem=4.2/32.7GB | cost=$16.81/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 03:09 (132.2h) | Phase3: SKIPPED | SFT: Oct 27 07:52 (136.9h) | DPO (Final): Oct 27 08:26 (137.5h) [2026-10-21 14:59:00][INFO][training.pretrain_engine] Starting global step 206 (accumulating 32 micro-batches)... [2026-10-21 14:59:00][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 14:59:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0918 [2026-10-21 14:59:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8865 [2026-10-21 14:59:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7978 [2026-10-21 14:59:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5956 [2026-10-21 14:59:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4747 [2026-10-21 14:59:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1296 [2026-10-21 14:59:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8041 [2026-10-21 14:59:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1148 [2026-10-21 14:59:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3276 [2026-10-21 14:59:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5487 [2026-10-21 14:59:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8719 [2026-10-21 14:59:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.1674 [2026-10-21 15:00:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1023 [2026-10-21 15:00:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3087 [2026-10-21 15:00:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2539 [2026-10-21 15:00:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2345 [2026-10-21 15:00:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9315 [2026-10-21 15:00:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4618 [2026-10-21 15:00:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3421 [2026-10-21 15:00:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4057 [2026-10-21 15:00:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2113 [2026-10-21 15:00:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4909 [2026-10-21 15:00:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3913 [2026-10-21 15:00:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4110 [2026-10-21 15:01:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2822 [2026-10-21 15:01:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3969 [2026-10-21 15:01:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8416 [2026-10-21 15:01:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0288 [2026-10-21 15:01:18][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,632 | MFU: 1.8% [2026-10-21 15:01:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0648 [2026-10-21 15:01:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1949 [2026-10-21 15:01:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4527 [2026-10-21 15:01:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6723 [2026-10-21 15:01:23][INFO][training.pretrain_engine] [Ph2|Step 206] loss=5.2343 | ppl=187.59 | aux=0.02507 | lr=2.99e-04 | gnorm=0.789 | tok/s=29,387 | MFU=7.1% | step=142727ms | tokens=1.018B | mem=4.2/32.7GB | cost=$16.89/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 03:11 (132.2h) | Phase3: SKIPPED | SFT: Oct 27 07:54 (136.9h) | DPO (Final): Oct 27 08:28 (137.5h) [2026-10-21 15:01:23][INFO][training.pretrain_engine] Starting global step 207 (accumulating 32 micro-batches)... [2026-10-21 15:01:23][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 15:01:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0766 [2026-10-21 15:01:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7461 [2026-10-21 15:01:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8137 [2026-10-21 15:01:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1935 [2026-10-21 15:01:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7730 [2026-10-21 15:01:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2446 [2026-10-21 15:01:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5499 [2026-10-21 15:01:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4878 [2026-10-21 15:02:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1356 [2026-10-21 15:02:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5706 [2026-10-21 15:02:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9946 [2026-10-21 15:02:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2270 [2026-10-21 15:02:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4095 [2026-10-21 15:02:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2477 [2026-10-21 15:02:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3518 [2026-10-21 15:02:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4246 [2026-10-21 15:02:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4332 [2026-10-21 15:02:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8462 [2026-10-21 15:02:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9421 [2026-10-21 15:02:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4693 [2026-10-21 15:03:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8263 [2026-10-21 15:03:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5757 [2026-10-21 15:03:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8528 [2026-10-21 15:03:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8294 [2026-10-21 15:03:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0362 [2026-10-21 15:03:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0284 [2026-10-21 15:03:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8576 [2026-10-21 15:03:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1332 [2026-10-21 15:03:41][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,614 | MFU: 1.8% [2026-10-21 15:03:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1475 [2026-10-21 15:03:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.0843 [2026-10-21 15:03:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1487 [2026-10-21 15:03:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5655 [2026-10-21 15:03:46][INFO][training.pretrain_engine] [Ph2|Step 207] loss=5.1745 | ppl=176.71 | aux=0.02507 | lr=2.99e-04 | gnorm=0.672 | tok/s=29,387 | MFU=7.1% | step=142725ms | tokens=1.019B | mem=4.2/32.7GB | cost=$16.97/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 03:12 (132.2h) | Phase3: SKIPPED | SFT: Oct 27 07:56 (136.9h) | DPO (Final): Oct 27 08:30 (137.4h) [2026-10-21 15:03:50][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step207 | Tokens: 1.019B | Spent: $16.97 | Remaining: $983.03 [2026-10-21 15:03:50][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step207 [2026-10-21 15:03:50][INFO][training.pretrain_engine] Starting global step 208 (accumulating 32 micro-batches)... [2026-10-21 15:03:50][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 15:03:50][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 15:03:50][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 15:03:50][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 15:03:50][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step207 to Phase-Technologies/forge-500m-pretrain (path: phase2_step207)... [2026-10-21 15:03:50][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 15:04:02][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 15:04:02][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step207 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 15:04:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1439 [2026-10-21 15:04:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5954 [2026-10-21 15:04:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3561 [2026-10-21 15:04:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3332 [2026-10-21 15:04:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4129 [2026-10-21 15:04:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3294 [2026-10-21 15:04:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1644 [2026-10-21 15:04:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2155 [2026-10-21 15:04:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5284 [2026-10-21 15:04:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5329 [2026-10-21 15:04:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7250 [2026-10-21 15:04:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4157 [2026-10-21 15:04:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5549 [2026-10-21 15:04:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3506 [2026-10-21 15:04:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5618 [2026-10-21 15:05:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4721 [2026-10-21 15:05:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0106 [2026-10-21 15:05:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0988 [2026-10-21 15:05:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9528 [2026-10-21 15:05:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9870 [2026-10-21 15:05:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8829 [2026-10-21 15:05:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1246 [2026-10-21 15:05:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1136 [2026-10-21 15:05:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6689 [2026-10-21 15:05:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6420 [2026-10-21 15:05:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0028 [2026-10-21 15:05:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2807 [2026-10-21 15:05:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6048 [2026-10-21 15:06:07][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,624 | MFU: 1.8% [2026-10-21 15:06:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4983 [2026-10-21 15:06:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7327 [2026-10-21 15:06:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7776 [2026-10-21 15:06:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5683 [2026-10-21 15:06:12][INFO][training.pretrain_engine] [Ph2|Step 208] loss=5.1850 | ppl=178.57 | aux=0.02508 | lr=2.99e-04 | gnorm=0.629 | tok/s=29,390 | MFU=7.1% | step=142711ms | tokens=1.020B | mem=4.2/32.7GB | cost=$17.05/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 03:13 (132.1h) | Phase3: SKIPPED | SFT: Oct 27 07:57 (136.9h) | DPO (Final): Oct 27 08:31 (137.4h) [2026-10-21 15:06:12][INFO][training.pretrain_engine] Starting global step 209 (accumulating 32 micro-batches)... [2026-10-21 15:06:12][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 15:06:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5917 [2026-10-21 15:06:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6160 [2026-10-21 15:06:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2217 [2026-10-21 15:06:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4434 [2026-10-21 15:06:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5407 [2026-10-21 15:06:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3749 [2026-10-21 15:06:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2761 [2026-10-21 15:06:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4360 [2026-10-21 15:07:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1295 [2026-10-21 15:07:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8846 [2026-10-21 15:07:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9045 [2026-10-21 15:07:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8296 [2026-10-21 15:07:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4594 [2026-10-21 15:07:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4778 [2026-10-21 15:07:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2873 [2026-10-21 15:07:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8970 [2026-10-21 15:07:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3011 [2026-10-21 15:07:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4232 [2026-10-21 15:07:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3408 [2026-10-21 15:07:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5371 [2026-10-21 15:07:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2903 [2026-10-21 15:07:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7322 [2026-10-21 15:07:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1421 [2026-10-21 15:07:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2038 [2026-10-21 15:08:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9934 [2026-10-21 15:08:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8957 [2026-10-21 15:08:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9921 [2026-10-21 15:08:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4528 [2026-10-21 15:08:29][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,621 | MFU: 1.8% [2026-10-21 15:08:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1309 [2026-10-21 15:08:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3201 [2026-10-21 15:08:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4996 [2026-10-21 15:08:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1158 [2026-10-21 15:08:34][INFO][training.pretrain_engine] [Ph2|Step 209] loss=5.1271 | ppl=168.53 | aux=0.02508 | lr=2.99e-04 | gnorm=0.656 | tok/s=29,389 | MFU=7.1% | step=142718ms | tokens=1.021B | mem=4.2/32.7GB | cost=$17.13/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 03:16 (132.1h) | Phase3: SKIPPED | SFT: Oct 27 07:59 (136.8h) | DPO (Final): Oct 27 08:33 (137.4h) [2026-10-21 15:08:34][INFO][training.pretrain_engine] Starting global step 210 (accumulating 32 micro-batches)... [2026-10-21 15:08:34][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 15:08:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9226 [2026-10-21 15:08:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1940 [2026-10-21 15:08:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3048 [2026-10-21 15:08:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9954 [2026-10-21 15:09:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1827 [2026-10-21 15:09:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9854 [2026-10-21 15:09:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1753 [2026-10-21 15:09:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5028 [2026-10-21 15:09:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2088 [2026-10-21 15:09:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2428 [2026-10-21 15:09:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7686 [2026-10-21 15:09:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1283 [2026-10-21 15:09:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5393 [2026-10-21 15:09:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8120 [2026-10-21 15:09:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6829 [2026-10-21 15:09:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6706 [2026-10-21 15:10:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3739 [2026-10-21 15:10:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0764 [2026-10-21 15:10:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6027 [2026-10-21 15:10:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3827 [2026-10-21 15:10:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8709 [2026-10-21 15:10:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5134 [2026-10-21 15:10:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9112 [2026-10-21 15:10:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0068 [2026-10-21 15:10:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0125 [2026-10-21 15:10:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2359 [2026-10-21 15:10:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1491 [2026-10-21 15:10:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2878 [2026-10-21 15:10:52][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,613 | MFU: 1.8% [2026-10-21 15:10:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2599 [2026-10-21 15:10:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3554 [2026-10-21 15:10:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5707 [2026-10-21 15:10:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2832 [2026-10-21 15:10:57][INFO][training.pretrain_engine] [Ph2|Step 210] loss=5.2166 | ppl=184.31 | aux=0.02506 | lr=2.99e-04 | gnorm=0.559 | tok/s=29,390 | MFU=7.1% | step=142714ms | tokens=1.022B | mem=4.2/32.7GB | cost=$17.22/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 03:17 (132.1h) | Phase3: SKIPPED | SFT: Oct 27 08:01 (136.8h) | DPO (Final): Oct 27 08:35 (137.4h) [2026-10-21 15:10:57][INFO][training.pretrain_engine] Starting global step 211 (accumulating 32 micro-batches)... [2026-10-21 15:10:57][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 15:11:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1131 [2026-10-21 15:11:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2803 [2026-10-21 15:11:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4955 [2026-10-21 15:11:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2272 [2026-10-21 15:11:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1661 [2026-10-21 15:11:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4785 [2026-10-21 15:11:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9795 [2026-10-21 15:11:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2590 [2026-10-21 15:11:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3550 [2026-10-21 15:11:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4824 [2026-10-21 15:11:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0501 [2026-10-21 15:11:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8791 [2026-10-21 15:12:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4614 [2026-10-21 15:12:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2548 [2026-10-21 15:12:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1830 [2026-10-21 15:12:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6586 [2026-10-21 15:12:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1311 [2026-10-21 15:12:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2942 [2026-10-21 15:12:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3961 [2026-10-21 15:12:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0616 [2026-10-21 15:12:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3613 [2026-10-21 15:12:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9590 [2026-10-21 15:12:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4183 [2026-10-21 15:12:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1919 [2026-10-21 15:13:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4675 [2026-10-21 15:13:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5550 [2026-10-21 15:13:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3872 [2026-10-21 15:13:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2880 [2026-10-21 15:13:15][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,633 | MFU: 1.8% [2026-10-21 15:13:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2543 [2026-10-21 15:13:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0086 [2026-10-21 15:13:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3510 [2026-10-21 15:13:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2858 [2026-10-21 15:13:20][INFO][training.pretrain_engine] [Ph2|Step 211] loss=5.2501 | ppl=190.59 | aux=0.02506 | lr=2.99e-04 | gnorm=0.551 | tok/s=29,391 | MFU=7.1% | step=142709ms | tokens=1.023B | mem=4.2/32.7GB | cost=$17.30/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 03:19 (132.1h) | Phase3: SKIPPED | SFT: Oct 27 08:02 (136.8h) | DPO (Final): Oct 27 08:36 (137.4h) [2026-10-21 15:13:20][INFO][training.pretrain_engine] Starting global step 212 (accumulating 32 micro-batches)... [2026-10-21 15:13:20][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 15:13:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9905 [2026-10-21 15:13:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3971 [2026-10-21 15:13:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0334 [2026-10-21 15:13:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6562 [2026-10-21 15:13:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1860 [2026-10-21 15:13:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9758 [2026-10-21 15:13:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3330 [2026-10-21 15:13:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2648 [2026-10-21 15:14:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9090 [2026-10-21 15:14:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2478 [2026-10-21 15:14:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2361 [2026-10-21 15:14:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2245 [2026-10-21 15:14:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9694 [2026-10-21 15:14:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3932 [2026-10-21 15:14:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7117 [2026-10-21 15:14:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9402 [2026-10-21 15:14:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1871 [2026-10-21 15:14:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1341 [2026-10-21 15:14:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8476 [2026-10-21 15:14:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5955 [2026-10-21 15:15:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2304 [2026-10-21 15:15:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0387 [2026-10-21 15:15:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0428 [2026-10-21 15:15:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4026 [2026-10-21 15:15:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8805 [2026-10-21 15:15:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7819 [2026-10-21 15:15:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4161 [2026-10-21 15:15:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5341 [2026-10-21 15:15:37][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,633 | MFU: 1.8% [2026-10-21 15:15:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6478 [2026-10-21 15:15:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7311 [2026-10-21 15:15:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6837 [2026-10-21 15:15:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5021 [2026-10-21 15:15:43][INFO][training.pretrain_engine] [Ph2|Step 212] loss=5.2474 | ppl=190.07 | aux=0.02506 | lr=2.99e-04 | gnorm=0.547 | tok/s=29,391 | MFU=7.1% | step=142706ms | tokens=1.024B | mem=4.2/32.7GB | cost=$17.38/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 03:20 (132.1h) | Phase3: SKIPPED | SFT: Oct 27 08:04 (136.8h) | DPO (Final): Oct 27 08:38 (137.4h) [2026-10-21 15:15:43][INFO][training.pretrain_engine] Starting global step 213 (accumulating 32 micro-batches)... [2026-10-21 15:15:43][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 15:16:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1981 [2026-10-21 15:16:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9593 [2026-10-21 15:16:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2090 [2026-10-21 15:16:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0492 [2026-10-21 15:16:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8927 [2026-10-21 15:16:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1828 [2026-10-21 15:16:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1303 [2026-10-21 15:16:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8772 [2026-10-21 15:16:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7263 [2026-10-21 15:16:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1647 [2026-10-21 15:16:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3529 [2026-10-21 15:16:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5468 [2026-10-21 15:16:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8946 [2026-10-21 15:16:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8889 [2026-10-21 15:16:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7481 [2026-10-21 15:16:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3029 [2026-10-21 15:17:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5491 [2026-10-21 15:17:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4542 [2026-10-21 15:17:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3614 [2026-10-21 15:17:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1418 [2026-10-21 15:17:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8218 [2026-10-21 15:17:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1808 [2026-10-21 15:17:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0765 [2026-10-21 15:17:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5293 [2026-10-21 15:17:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1278 [2026-10-21 15:17:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2107 [2026-10-21 15:17:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2434 [2026-10-21 15:17:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0096 [2026-10-21 15:18:00][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,621 | MFU: 1.8% [2026-10-21 15:18:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4940 [2026-10-21 15:18:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7517 [2026-10-21 15:18:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5029 [2026-10-21 15:18:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2221 [2026-10-21 15:18:06][INFO][training.pretrain_engine] [Ph2|Step 213] loss=5.1775 | ppl=177.23 | aux=0.02506 | lr=2.99e-04 | gnorm=0.465 | tok/s=29,391 | MFU=7.1% | step=142705ms | tokens=1.026B | mem=4.2/32.7GB | cost=$17.46/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 03:22 (132.1h) | Phase3: SKIPPED | SFT: Oct 27 08:05 (136.8h) | DPO (Final): Oct 27 08:40 (137.4h) [2026-10-21 15:18:06][INFO][training.pretrain_engine] Starting global step 214 (accumulating 32 micro-batches)... [2026-10-21 15:18:06][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 15:18:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9958 [2026-10-21 15:18:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1941 [2026-10-21 15:18:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1980 [2026-10-21 15:18:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9286 [2026-10-21 15:18:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3193 [2026-10-21 15:18:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5059 [2026-10-21 15:18:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3404 [2026-10-21 15:18:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5727 [2026-10-21 15:18:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6055 [2026-10-21 15:18:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2650 [2026-10-21 15:18:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1965 [2026-10-21 15:18:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4659 [2026-10-21 15:19:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9055 [2026-10-21 15:19:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7642 [2026-10-21 15:19:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8959 [2026-10-21 15:19:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7246 [2026-10-21 15:19:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2937 [2026-10-21 15:19:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1659 [2026-10-21 15:19:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1794 [2026-10-21 15:19:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7455 [2026-10-21 15:19:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4187 [2026-10-21 15:19:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3134 [2026-10-21 15:19:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3296 [2026-10-21 15:19:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7245 [2026-10-21 15:20:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3970 [2026-10-21 15:20:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1210 [2026-10-21 15:20:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3536 [2026-10-21 15:20:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0322 [2026-10-21 15:20:23][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,625 | MFU: 1.8% [2026-10-21 15:20:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1246 [2026-10-21 15:20:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5178 [2026-10-21 15:20:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9648 [2026-10-21 15:20:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2779 [2026-10-21 15:20:29][INFO][training.pretrain_engine] [Ph2|Step 214] loss=5.0611 | ppl=157.76 | aux=0.02507 | lr=2.99e-04 | gnorm=0.531 | tok/s=29,391 | MFU=7.1% | step=142708ms | tokens=1.027B | mem=4.2/32.7GB | cost=$17.54/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 03:24 (132.1h) | Phase3: SKIPPED | SFT: Oct 27 08:07 (136.8h) | DPO (Final): Oct 27 08:41 (137.4h) [2026-10-21 15:20:29][INFO][training.pretrain_engine] Starting global step 215 (accumulating 32 micro-batches)... [2026-10-21 15:20:29][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 15:20:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9677 [2026-10-21 15:20:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1782 [2026-10-21 15:20:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1495 [2026-10-21 15:20:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3082 [2026-10-21 15:21:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9163 [2026-10-21 15:21:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3039 [2026-10-21 15:21:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6273 [2026-10-21 15:21:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.9194 [2026-10-21 15:21:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7184 [2026-10-21 15:21:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0475 [2026-10-21 15:21:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5849 [2026-10-21 15:21:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3579 [2026-10-21 15:21:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5097 [2026-10-21 15:21:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6390 [2026-10-21 15:21:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5232 [2026-10-21 15:21:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6977 [2026-10-21 15:21:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3522 [2026-10-21 15:21:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3290 [2026-10-21 15:21:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1418 [2026-10-21 15:21:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6339 [2026-10-21 15:22:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2902 [2026-10-21 15:22:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6822 [2026-10-21 15:22:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9149 [2026-10-21 15:22:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6279 [2026-10-21 15:22:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7833 [2026-10-21 15:22:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5209 [2026-10-21 15:22:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3589 [2026-10-21 15:22:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2378 [2026-10-21 15:22:46][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,621 | MFU: 1.8% [2026-10-21 15:22:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5578 [2026-10-21 15:22:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3408 [2026-10-21 15:22:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6147 [2026-10-21 15:22:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2366 [2026-10-21 15:22:51][INFO][training.pretrain_engine] [Ph2|Step 215] loss=5.2413 | ppl=188.92 | aux=0.02507 | lr=2.99e-04 | gnorm=0.477 | tok/s=29,391 | MFU=7.1% | step=142706ms | tokens=1.028B | mem=4.2/32.7GB | cost=$17.62/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 03:26 (132.1h) | Phase3: SKIPPED | SFT: Oct 27 08:09 (136.8h) | DPO (Final): Oct 27 08:43 (137.3h) [2026-10-21 15:22:51][INFO][training.pretrain_engine] Starting global step 216 (accumulating 32 micro-batches)... [2026-10-21 15:22:51][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 15:23:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2346 [2026-10-21 15:23:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4684 [2026-10-21 15:23:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6240 [2026-10-21 15:23:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9590 [2026-10-21 15:23:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4773 [2026-10-21 15:23:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7586 [2026-10-21 15:23:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1672 [2026-10-21 15:23:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0469 [2026-10-21 15:23:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2441 [2026-10-21 15:23:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8602 [2026-10-21 15:23:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8275 [2026-10-21 15:23:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3180 [2026-10-21 15:24:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7102 [2026-10-21 15:24:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1346 [2026-10-21 15:24:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2889 [2026-10-21 15:24:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1418 [2026-10-21 15:24:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8893 [2026-10-21 15:24:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7161 [2026-10-21 15:24:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0496 [2026-10-21 15:24:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1412 [2026-10-21 15:24:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2247 [2026-10-21 15:24:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7349 [2026-10-21 15:24:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1435 [2026-10-21 15:24:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4885 [2026-10-21 15:24:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4344 [2026-10-21 15:24:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4246 [2026-10-21 15:24:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4891 [2026-10-21 15:24:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6688 [2026-10-21 15:25:09][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,606 | MFU: 1.8% [2026-10-21 15:25:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6959 [2026-10-21 15:25:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8919 [2026-10-21 15:25:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3743 [2026-10-21 15:25:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4654 [2026-10-21 15:25:14][INFO][training.pretrain_engine] [Ph2|Step 216] loss=5.2939 | ppl=199.12 | aux=0.02508 | lr=2.99e-04 | gnorm=0.484 | tok/s=29,392 | MFU=7.1% | step=142704ms | tokens=1.029B | mem=4.2/32.7GB | cost=$17.71/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 03:27 (132.0h) | Phase3: SKIPPED | SFT: Oct 27 08:11 (136.8h) | DPO (Final): Oct 27 08:45 (137.3h) [2026-10-21 15:25:17][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step216 | Tokens: 1.029B | Spent: $17.71 | Remaining: $982.29 [2026-10-21 15:25:17][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step216 [2026-10-21 15:25:17][INFO][training.pretrain_engine] Starting global step 217 (accumulating 32 micro-batches)... [2026-10-21 15:25:17][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 15:25:17][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 15:25:17][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 15:25:17][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 15:25:17][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step216 to Phase-Technologies/forge-500m-pretrain (path: phase2_step216)... [2026-10-21 15:25:18][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 15:25:30][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 15:25:30][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step216 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 15:25:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2748 [2026-10-21 15:25:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1429 [2026-10-21 15:25:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0302 [2026-10-21 15:25:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0574 [2026-10-21 15:25:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5219 [2026-10-21 15:25:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5134 [2026-10-21 15:25:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4576 [2026-10-21 15:25:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2495 [2026-10-21 15:26:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7096 [2026-10-21 15:26:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4319 [2026-10-21 15:26:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4728 [2026-10-21 15:26:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5719 [2026-10-21 15:26:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5689 [2026-10-21 15:26:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8416 [2026-10-21 15:26:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8089 [2026-10-21 15:26:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9532 [2026-10-21 15:26:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3665 [2026-10-21 15:26:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3760 [2026-10-21 15:26:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8342 [2026-10-21 15:26:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6155 [2026-10-21 15:27:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0468 [2026-10-21 15:27:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0042 [2026-10-21 15:27:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8551 [2026-10-21 15:27:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6125 [2026-10-21 15:27:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1734 [2026-10-21 15:27:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3481 [2026-10-21 15:27:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9866 [2026-10-21 15:27:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8476 [2026-10-21 15:27:35][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,632 | MFU: 1.8% [2026-10-21 15:27:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0998 [2026-10-21 15:27:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3665 [2026-10-21 15:27:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0173 [2026-10-21 15:27:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1836 [2026-10-21 15:27:39][INFO][training.pretrain_engine] [Ph2|Step 217] loss=5.2009 | ppl=181.43 | aux=0.02506 | lr=2.99e-04 | gnorm=0.482 | tok/s=29,394 | MFU=7.1% | step=142691ms | tokens=1.030B | mem=4.2/32.7GB | cost=$17.79/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 03:28 (132.0h) | Phase3: SKIPPED | SFT: Oct 27 08:12 (136.7h) | DPO (Final): Oct 27 08:46 (137.3h) [2026-10-21 15:27:39][INFO][training.pretrain_engine] Starting global step 218 (accumulating 32 micro-batches)... [2026-10-21 15:27:39][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 15:27:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9003 [2026-10-21 15:27:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0537 [2026-10-21 15:27:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3495 [2026-10-21 15:27:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7358 [2026-10-21 15:28:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2387 [2026-10-21 15:28:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9872 [2026-10-21 15:28:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2352 [2026-10-21 15:28:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3187 [2026-10-21 15:28:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9642 [2026-10-21 15:28:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8037 [2026-10-21 15:28:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5713 [2026-10-21 15:28:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0416 [2026-10-21 15:28:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3671 [2026-10-21 15:28:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4054 [2026-10-21 15:28:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1304 [2026-10-21 15:28:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1451 [2026-10-21 15:29:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0330 [2026-10-21 15:29:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4531 [2026-10-21 15:29:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1026 [2026-10-21 15:29:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0041 [2026-10-21 15:29:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2733 [2026-10-21 15:29:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9321 [2026-10-21 15:29:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7784 [2026-10-21 15:29:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3862 [2026-10-21 15:29:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5760 [2026-10-21 15:29:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4269 [2026-10-21 15:29:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6752 [2026-10-21 15:29:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8803 [2026-10-21 15:29:57][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,624 | MFU: 1.8% [2026-10-21 15:30:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6569 [2026-10-21 15:30:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4569 [2026-10-21 15:30:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2962 [2026-10-21 15:30:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5453 [2026-10-21 15:30:02][INFO][training.pretrain_engine] [Ph2|Step 218] loss=5.0964 | ppl=163.44 | aux=0.02505 | lr=2.99e-04 | gnorm=0.508 | tok/s=29,394 | MFU=7.1% | step=142695ms | tokens=1.031B | mem=4.2/32.7GB | cost=$17.87/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 03:30 (132.0h) | Phase3: SKIPPED | SFT: Oct 27 08:14 (136.7h) | DPO (Final): Oct 27 08:48 (137.3h) [2026-10-21 15:30:02][INFO][training.pretrain_engine] Starting global step 219 (accumulating 32 micro-batches)... [2026-10-21 15:30:02][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 15:30:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7587 [2026-10-21 15:30:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0469 [2026-10-21 15:30:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0334 [2026-10-21 15:30:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0906 [2026-10-21 15:30:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2866 [2026-10-21 15:30:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4955 [2026-10-21 15:30:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3304 [2026-10-21 15:30:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5960 [2026-10-21 15:30:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0897 [2026-10-21 15:30:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4826 [2026-10-21 15:30:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4200 [2026-10-21 15:30:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8528 [2026-10-21 15:31:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3305 [2026-10-21 15:31:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8743 [2026-10-21 15:31:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1565 [2026-10-21 15:31:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1917 [2026-10-21 15:31:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0791 [2026-10-21 15:31:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5024 [2026-10-21 15:31:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2302 [2026-10-21 15:31:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3053 [2026-10-21 15:31:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6305 [2026-10-21 15:31:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8160 [2026-10-21 15:31:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5541 [2026-10-21 15:31:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6187 [2026-10-21 15:32:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2087 [2026-10-21 15:32:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5371 [2026-10-21 15:32:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2695 [2026-10-21 15:32:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1700 [2026-10-21 15:32:20][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,614 | MFU: 1.8% [2026-10-21 15:32:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8361 [2026-10-21 15:32:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1580 [2026-10-21 15:32:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2487 [2026-10-21 15:32:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2466 [2026-10-21 15:32:25][INFO][training.pretrain_engine] [Ph2|Step 219] loss=5.2266 | ppl=186.16 | aux=0.02506 | lr=2.99e-04 | gnorm=0.578 | tok/s=29,394 | MFU=7.1% | step=142694ms | tokens=1.032B | mem=4.2/32.7GB | cost=$17.95/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 03:32 (132.0h) | Phase3: SKIPPED | SFT: Oct 27 08:16 (136.7h) | DPO (Final): Oct 27 08:50 (137.3h) [2026-10-21 15:32:25][INFO][training.pretrain_engine] Starting global step 220 (accumulating 32 micro-batches)... [2026-10-21 15:32:25][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 15:32:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9420 [2026-10-21 15:32:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8474 [2026-10-21 15:32:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9409 [2026-10-21 15:32:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8757 [2026-10-21 15:33:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7446 [2026-10-21 15:33:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2372 [2026-10-21 15:33:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5796 [2026-10-21 15:33:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9405 [2026-10-21 15:33:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2291 [2026-10-21 15:33:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0415 [2026-10-21 15:33:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0832 [2026-10-21 15:33:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7969 [2026-10-21 15:33:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2765 [2026-10-21 15:33:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1371 [2026-10-21 15:33:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0639 [2026-10-21 15:33:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0286 [2026-10-21 15:33:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9037 [2026-10-21 15:33:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9559 [2026-10-21 15:33:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1401 [2026-10-21 15:33:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8859 [2026-10-21 15:34:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0060 [2026-10-21 15:34:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3193 [2026-10-21 15:34:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7466 [2026-10-21 15:34:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4482 [2026-10-21 15:34:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1076 [2026-10-21 15:34:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7511 [2026-10-21 15:34:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1710 [2026-10-21 15:34:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5160 [2026-10-21 15:34:42][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,646 | MFU: 1.8% [2026-10-21 15:34:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0449 [2026-10-21 15:34:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3298 [2026-10-21 15:34:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2772 [2026-10-21 15:34:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9765 [2026-10-21 15:34:47][INFO][training.pretrain_engine] [Ph2|Step 220] loss=5.1408 | ppl=170.85 | aux=0.02506 | lr=2.99e-04 | gnorm=0.594 | tok/s=29,394 | MFU=7.1% | step=142694ms | tokens=1.033B | mem=4.2/32.7GB | cost=$18.03/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 03:34 (132.0h) | Phase3: SKIPPED | SFT: Oct 27 08:17 (136.7h) | DPO (Final): Oct 27 08:51 (137.3h) [2026-10-21 15:34:47][INFO][training.pretrain_engine] Starting global step 221 (accumulating 32 micro-batches)... [2026-10-21 15:34:47][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 15:35:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2629 [2026-10-21 15:35:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2013 [2026-10-21 15:35:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4609 [2026-10-21 15:35:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3201 [2026-10-21 15:35:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7611 [2026-10-21 15:35:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2571 [2026-10-21 15:35:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.9001 [2026-10-21 15:35:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1874 [2026-10-21 15:35:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8075 [2026-10-21 15:35:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8689 [2026-10-21 15:35:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3798 [2026-10-21 15:35:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1438 [2026-10-21 15:35:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1266 [2026-10-21 15:35:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1240 [2026-10-21 15:35:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3554 [2026-10-21 15:35:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3134 [2026-10-21 15:36:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8311 [2026-10-21 15:36:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8452 [2026-10-21 15:36:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6048 [2026-10-21 15:36:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8589 [2026-10-21 15:36:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8550 [2026-10-21 15:36:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2900 [2026-10-21 15:36:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1205 [2026-10-21 15:36:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5947 [2026-10-21 15:36:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0725 [2026-10-21 15:36:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0311 [2026-10-21 15:36:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3729 [2026-10-21 15:36:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8815 [2026-10-21 15:37:05][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,632 | MFU: 1.8% [2026-10-21 15:37:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3113 [2026-10-21 15:37:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3444 [2026-10-21 15:37:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4153 [2026-10-21 15:37:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0090 [2026-10-21 15:37:10][INFO][training.pretrain_engine] [Ph2|Step 221] loss=5.1292 | ppl=168.87 | aux=0.02507 | lr=2.99e-04 | gnorm=0.789 | tok/s=29,393 | MFU=7.1% | step=142696ms | tokens=1.034B | mem=4.2/32.7GB | cost=$18.12/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 03:36 (132.0h) | Phase3: SKIPPED | SFT: Oct 27 08:19 (136.7h) | DPO (Final): Oct 27 08:53 (137.3h) [2026-10-21 15:37:10][INFO][training.pretrain_engine] Starting global step 222 (accumulating 32 micro-batches)... [2026-10-21 15:37:10][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 15:37:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1318 [2026-10-21 15:37:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3635 [2026-10-21 15:37:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0070 [2026-10-21 15:37:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1728 [2026-10-21 15:37:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3431 [2026-10-21 15:37:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9710 [2026-10-21 15:37:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1223 [2026-10-21 15:37:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6743 [2026-10-21 15:38:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8730 [2026-10-21 15:38:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2593 [2026-10-21 15:38:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7180 [2026-10-21 15:38:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4144 [2026-10-21 15:38:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5627 [2026-10-21 15:38:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9900 [2026-10-21 15:38:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7721 [2026-10-21 15:38:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4462 [2026-10-21 15:38:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0988 [2026-10-21 15:38:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1195 [2026-10-21 15:38:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3825 [2026-10-21 15:38:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0928 [2026-10-21 15:38:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0002 [2026-10-21 15:38:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9494 [2026-10-21 15:38:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5268 [2026-10-21 15:38:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1427 [2026-10-21 15:39:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1138 [2026-10-21 15:39:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.1391 [2026-10-21 15:39:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4504 [2026-10-21 15:39:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0106 [2026-10-21 15:39:28][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,602 | MFU: 1.8% [2026-10-21 15:39:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5375 [2026-10-21 15:39:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4753 [2026-10-21 15:39:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6308 [2026-10-21 15:39:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4654 [2026-10-21 15:39:33][INFO][training.pretrain_engine] [Ph2|Step 222] loss=5.1282 | ppl=168.71 | aux=0.02506 | lr=2.99e-04 | gnorm=0.969 | tok/s=29,393 | MFU=7.1% | step=142698ms | tokens=1.035B | mem=4.2/32.7GB | cost=$18.20/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 03:38 (132.0h) | Phase3: SKIPPED | SFT: Oct 27 08:21 (136.7h) | DPO (Final): Oct 27 08:55 (137.3h) [2026-10-21 15:39:33][INFO][training.pretrain_engine] Starting global step 223 (accumulating 32 micro-batches)... [2026-10-21 15:39:33][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 15:39:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6001 [2026-10-21 15:39:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1607 [2026-10-21 15:39:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4385 [2026-10-21 15:39:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0717 [2026-10-21 15:40:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2241 [2026-10-21 15:40:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0864 [2026-10-21 15:40:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9148 [2026-10-21 15:40:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9071 [2026-10-21 15:40:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5280 [2026-10-21 15:40:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1719 [2026-10-21 15:40:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4388 [2026-10-21 15:40:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3563 [2026-10-21 15:40:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1437 [2026-10-21 15:40:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2082 [2026-10-21 15:40:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1166 [2026-10-21 15:40:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9424 [2026-10-21 15:41:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5361 [2026-10-21 15:41:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8894 [2026-10-21 15:41:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3940 [2026-10-21 15:41:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4280 [2026-10-21 15:41:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9559 [2026-10-21 15:41:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1724 [2026-10-21 15:41:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1038 [2026-10-21 15:41:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8874 [2026-10-21 15:41:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4524 [2026-10-21 15:41:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2350 [2026-10-21 15:41:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1288 [2026-10-21 15:41:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4236 [2026-10-21 15:41:51][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,602 | MFU: 1.8% [2026-10-21 15:41:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4746 [2026-10-21 15:41:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7807 [2026-10-21 15:41:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8000 [2026-10-21 15:41:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6109 [2026-10-21 15:41:56][INFO][training.pretrain_engine] [Ph2|Step 223] loss=5.1177 | ppl=166.95 | aux=0.02506 | lr=2.99e-04 | gnorm=0.930 | tok/s=29,393 | MFU=7.1% | step=142698ms | tokens=1.036B | mem=4.2/32.7GB | cost=$18.28/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 03:39 (132.0h) | Phase3: SKIPPED | SFT: Oct 27 08:23 (136.7h) | DPO (Final): Oct 27 08:57 (137.3h) [2026-10-21 15:41:56][INFO][training.pretrain_engine] Starting global step 224 (accumulating 32 micro-batches)... [2026-10-21 15:41:56][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 15:42:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4583 [2026-10-21 15:42:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7054 [2026-10-21 15:42:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9502 [2026-10-21 15:42:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7857 [2026-10-21 15:42:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1404 [2026-10-21 15:42:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4295 [2026-10-21 15:42:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0793 [2026-10-21 15:42:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9421 [2026-10-21 15:42:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1728 [2026-10-21 15:42:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8689 [2026-10-21 15:42:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0391 [2026-10-21 15:42:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8406 [2026-10-21 15:43:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3484 [2026-10-21 15:43:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0681 [2026-10-21 15:43:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8424 [2026-10-21 15:43:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3385 [2026-10-21 15:43:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9019 [2026-10-21 15:43:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0057 [2026-10-21 15:43:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6958 [2026-10-21 15:43:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8175 [2026-10-21 15:43:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0907 [2026-10-21 15:43:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3057 [2026-10-21 15:43:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2394 [2026-10-21 15:43:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3229 [2026-10-21 15:44:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6394 [2026-10-21 15:44:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5764 [2026-10-21 15:44:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5606 [2026-10-21 15:44:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1846 [2026-10-21 15:44:14][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,608 | MFU: 1.8% [2026-10-21 15:44:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.1962 [2026-10-21 15:44:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7937 [2026-10-21 15:44:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2027 [2026-10-21 15:44:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5426 [2026-10-21 15:44:19][INFO][training.pretrain_engine] [Ph2|Step 224] loss=5.1369 | ppl=170.19 | aux=0.02507 | lr=2.99e-04 | gnorm=0.906 | tok/s=29,392 | MFU=7.1% | step=142701ms | tokens=1.037B | mem=4.2/32.7GB | cost=$18.36/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 03:41 (132.0h) | Phase3: SKIPPED | SFT: Oct 27 08:25 (136.7h) | DPO (Final): Oct 27 08:59 (137.3h) [2026-10-21 15:44:19][INFO][training.pretrain_engine] Starting global step 225 (accumulating 32 micro-batches)... [2026-10-21 15:44:19][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 15:44:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0564 [2026-10-21 15:44:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2343 [2026-10-21 15:44:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1787 [2026-10-21 15:44:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9278 [2026-10-21 15:44:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5013 [2026-10-21 15:44:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7120 [2026-10-21 15:44:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3388 [2026-10-21 15:44:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2439 [2026-10-21 15:45:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2769 [2026-10-21 15:45:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7910 [2026-10-21 15:45:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3392 [2026-10-21 15:45:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7427 [2026-10-21 15:45:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4245 [2026-10-21 15:45:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7807 [2026-10-21 15:45:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1616 [2026-10-21 15:45:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1246 [2026-10-21 15:45:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3127 [2026-10-21 15:45:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2801 [2026-10-21 15:45:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9343 [2026-10-21 15:45:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2276 [2026-10-21 15:46:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9752 [2026-10-21 15:46:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1744 [2026-10-21 15:46:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7206 [2026-10-21 15:46:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0927 [2026-10-21 15:46:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8178 [2026-10-21 15:46:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4783 [2026-10-21 15:46:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4780 [2026-10-21 15:46:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6361 [2026-10-21 15:46:36][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,622 | MFU: 1.8% [2026-10-21 15:46:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2681 [2026-10-21 15:46:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9316 [2026-10-21 15:46:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6365 [2026-10-21 15:46:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1064 [2026-10-21 15:46:42][INFO][training.pretrain_engine] [Ph2|Step 225] loss=5.2001 | ppl=181.29 | aux=0.02507 | lr=2.99e-04 | gnorm=0.969 | tok/s=29,390 | MFU=7.1% | step=142711ms | tokens=1.038B | mem=4.2/32.7GB | cost=$18.44/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 03:44 (132.0h) | Phase3: SKIPPED | SFT: Oct 27 08:27 (136.7h) | DPO (Final): Oct 27 09:01 (137.3h) [2026-10-21 15:46:45][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step225 | Tokens: 1.038B | Spent: $18.45 | Remaining: $981.55 [2026-10-21 15:46:45][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step225 [2026-10-21 15:46:46][INFO][training.pretrain_engine] Starting global step 226 (accumulating 32 micro-batches)... [2026-10-21 15:46:46][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 15:46:46][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 15:46:46][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 15:46:46][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 15:46:46][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step225 to Phase-Technologies/forge-500m-pretrain (path: phase2_step225)... [2026-10-21 15:46:46][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 15:46:58][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 15:46:58][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step225 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 15:47:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5170 [2026-10-21 15:47:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4222 [2026-10-21 15:47:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4018 [2026-10-21 15:47:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9480 [2026-10-21 15:47:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9194 [2026-10-21 15:47:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5227 [2026-10-21 15:47:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6595 [2026-10-21 15:47:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5109 [2026-10-21 15:47:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0616 [2026-10-21 15:47:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0729 [2026-10-21 15:47:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7922 [2026-10-21 15:47:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6466 [2026-10-21 15:47:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3482 [2026-10-21 15:47:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2616 [2026-10-21 15:47:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4246 [2026-10-21 15:47:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8676 [2026-10-21 15:48:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2385 [2026-10-21 15:48:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2367 [2026-10-21 15:48:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5633 [2026-10-21 15:48:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8995 [2026-10-21 15:48:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7571 [2026-10-21 15:48:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3369 [2026-10-21 15:48:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3010 [2026-10-21 15:48:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1209 [2026-10-21 15:48:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4524 [2026-10-21 15:48:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2914 [2026-10-21 15:48:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0741 [2026-10-21 15:48:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2816 [2026-10-21 15:49:03][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,625 | MFU: 1.8% [2026-10-21 15:49:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9097 [2026-10-21 15:49:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8429 [2026-10-21 15:49:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2718 [2026-10-21 15:49:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2392 [2026-10-21 15:49:08][INFO][training.pretrain_engine] [Ph2|Step 226] loss=5.2028 | ppl=181.77 | aux=0.02507 | lr=2.99e-04 | gnorm=0.777 | tok/s=29,391 | MFU=7.1% | step=142706ms | tokens=1.039B | mem=4.2/32.7GB | cost=$18.53/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 03:45 (131.9h) | Phase3: SKIPPED | SFT: Oct 27 08:29 (136.7h) | DPO (Final): Oct 27 09:03 (137.2h) [2026-10-21 15:49:08][INFO][training.pretrain_engine] Starting global step 227 (accumulating 32 micro-batches)... [2026-10-21 15:49:08][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 15:49:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0923 [2026-10-21 15:49:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9037 [2026-10-21 15:49:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.5576 [2026-10-21 15:49:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7382 [2026-10-21 15:49:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5546 [2026-10-21 15:49:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2873 [2026-10-21 15:49:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9404 [2026-10-21 15:49:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1062 [2026-10-21 15:50:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9455 [2026-10-21 15:50:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2606 [2026-10-21 15:50:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1478 [2026-10-21 15:50:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.2000 [2026-10-21 15:50:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2757 [2026-10-21 15:50:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5558 [2026-10-21 15:50:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8915 [2026-10-21 15:50:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6871 [2026-10-21 15:50:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5922 [2026-10-21 15:50:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5821 [2026-10-21 15:50:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4369 [2026-10-21 15:50:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5261 [2026-10-21 15:50:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5835 [2026-10-21 15:50:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3593 [2026-10-21 15:50:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5615 [2026-10-21 15:50:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6285 [2026-10-21 15:51:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9965 [2026-10-21 15:51:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8984 [2026-10-21 15:51:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7761 [2026-10-21 15:51:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.4863 [2026-10-21 15:51:25][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,617 | MFU: 1.8% [2026-10-21 15:51:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2959 [2026-10-21 15:51:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9319 [2026-10-21 15:51:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6236 [2026-10-21 15:51:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9207 [2026-10-21 15:51:30][INFO][training.pretrain_engine] [Ph2|Step 227] loss=5.2600 | ppl=192.48 | aux=0.02506 | lr=2.99e-04 | gnorm=0.762 | tok/s=29,390 | MFU=7.1% | step=142712ms | tokens=1.040B | mem=4.2/32.7GB | cost=$18.61/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 03:47 (131.9h) | Phase3: SKIPPED | SFT: Oct 27 08:31 (136.7h) | DPO (Final): Oct 27 09:05 (137.2h) [2026-10-21 15:51:30][INFO][training.pretrain_engine] Starting global step 228 (accumulating 32 micro-batches)... [2026-10-21 15:51:30][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 15:51:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7922 [2026-10-21 15:51:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1867 [2026-10-21 15:51:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8592 [2026-10-21 15:51:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5919 [2026-10-21 15:52:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8150 [2026-10-21 15:52:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8495 [2026-10-21 15:52:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2580 [2026-10-21 15:52:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6335 [2026-10-21 15:52:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3958 [2026-10-21 15:52:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3933 [2026-10-21 15:52:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5675 [2026-10-21 15:52:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5131 [2026-10-21 15:52:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7157 [2026-10-21 15:52:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0912 [2026-10-21 15:52:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5168 [2026-10-21 15:52:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5764 [2026-10-21 15:52:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3641 [2026-10-21 15:52:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1317 [2026-10-21 15:52:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1151 [2026-10-21 15:53:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9180 [2026-10-21 15:53:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2534 [2026-10-21 15:53:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9180 [2026-10-21 15:53:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8668 [2026-10-21 15:53:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0814 [2026-10-21 15:53:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6603 [2026-10-21 15:53:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9926 [2026-10-21 15:53:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9367 [2026-10-21 15:53:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0716 [2026-10-21 15:53:48][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,623 | MFU: 1.8% [2026-10-21 15:53:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=3.8928 [2026-10-21 15:53:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7844 [2026-10-21 15:53:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5756 [2026-10-21 15:53:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5723 [2026-10-21 15:53:53][INFO][training.pretrain_engine] [Ph2|Step 228] loss=5.1925 | ppl=179.92 | aux=0.02506 | lr=2.99e-04 | gnorm=0.594 | tok/s=29,389 | MFU=7.1% | step=142716ms | tokens=1.041B | mem=4.2/32.7GB | cost=$18.69/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 03:49 (131.9h) | Phase3: SKIPPED | SFT: Oct 27 08:33 (136.7h) | DPO (Final): Oct 27 09:07 (137.2h) [2026-10-21 15:53:53][INFO][training.pretrain_engine] Starting global step 229 (accumulating 32 micro-batches)... [2026-10-21 15:53:53][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 15:54:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6372 [2026-10-21 15:54:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0057 [2026-10-21 15:54:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3471 [2026-10-21 15:54:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5158 [2026-10-21 15:54:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9976 [2026-10-21 15:54:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8602 [2026-10-21 15:54:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3780 [2026-10-21 15:54:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2958 [2026-10-21 15:54:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3299 [2026-10-21 15:54:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7572 [2026-10-21 15:54:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9504 [2026-10-21 15:54:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7918 [2026-10-21 15:55:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2007 [2026-10-21 15:55:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7804 [2026-10-21 15:55:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4243 [2026-10-21 15:55:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7755 [2026-10-21 15:55:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4817 [2026-10-21 15:55:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4859 [2026-10-21 15:55:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3173 [2026-10-21 15:55:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1119 [2026-10-21 15:55:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9555 [2026-10-21 15:55:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8996 [2026-10-21 15:55:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7248 [2026-10-21 15:55:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6246 [2026-10-21 15:55:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2836 [2026-10-21 15:55:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3472 [2026-10-21 15:55:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7373 [2026-10-21 15:55:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2816 [2026-10-21 15:56:11][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,626 | MFU: 1.8% [2026-10-21 15:56:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3600 [2026-10-21 15:56:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8221 [2026-10-21 15:56:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1548 [2026-10-21 15:56:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9695 [2026-10-21 15:56:16][INFO][training.pretrain_engine] [Ph2|Step 229] loss=5.2072 | ppl=182.58 | aux=0.02507 | lr=2.99e-04 | gnorm=0.664 | tok/s=29,388 | MFU=7.1% | step=142720ms | tokens=1.042B | mem=4.2/32.7GB | cost=$18.77/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 03:51 (131.9h) | Phase3: SKIPPED | SFT: Oct 27 08:35 (136.7h) | DPO (Final): Oct 27 09:09 (137.2h) [2026-10-21 15:56:16][INFO][training.pretrain_engine] Starting global step 230 (accumulating 32 micro-batches)... [2026-10-21 15:56:16][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 15:56:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0726 [2026-10-21 15:56:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9925 [2026-10-21 15:56:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9728 [2026-10-21 15:56:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2852 [2026-10-21 15:56:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3253 [2026-10-21 15:56:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1470 [2026-10-21 15:56:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8695 [2026-10-21 15:56:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0686 [2026-10-21 15:57:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4761 [2026-10-21 15:57:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1408 [2026-10-21 15:57:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3185 [2026-10-21 15:57:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9733 [2026-10-21 15:57:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0217 [2026-10-21 15:57:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1326 [2026-10-21 15:57:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3021 [2026-10-21 15:57:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9300 [2026-10-21 15:57:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5945 [2026-10-21 15:57:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3051 [2026-10-21 15:57:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0913 [2026-10-21 15:57:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7278 [2026-10-21 15:58:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3030 [2026-10-21 15:58:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4653 [2026-10-21 15:58:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4353 [2026-10-21 15:58:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3972 [2026-10-21 15:58:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5974 [2026-10-21 15:58:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3196 [2026-10-21 15:58:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8951 [2026-10-21 15:58:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9848 [2026-10-21 15:58:34][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,623 | MFU: 1.8% [2026-10-21 15:58:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6231 [2026-10-21 15:58:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3541 [2026-10-21 15:58:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0744 [2026-10-21 15:58:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6434 [2026-10-21 15:58:39][INFO][training.pretrain_engine] [Ph2|Step 230] loss=5.1997 | ppl=181.23 | aux=0.02506 | lr=2.99e-04 | gnorm=0.543 | tok/s=29,389 | MFU=7.1% | step=142716ms | tokens=1.043B | mem=4.2/32.7GB | cost=$18.85/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 03:53 (131.9h) | Phase3: SKIPPED | SFT: Oct 27 08:37 (136.6h) | DPO (Final): Oct 27 09:11 (137.2h) [2026-10-21 15:58:39][INFO][training.pretrain_engine] Starting global step 231 (accumulating 32 micro-batches)... [2026-10-21 15:58:39][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 15:58:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1570 [2026-10-21 15:58:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2939 [2026-10-21 15:58:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8538 [2026-10-21 15:58:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0999 [2026-10-21 15:59:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5860 [2026-10-21 15:59:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9418 [2026-10-21 15:59:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6995 [2026-10-21 15:59:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7336 [2026-10-21 15:59:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8821 [2026-10-21 15:59:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9508 [2026-10-21 15:59:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2054 [2026-10-21 15:59:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0186 [2026-10-21 15:59:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5460 [2026-10-21 15:59:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3096 [2026-10-21 15:59:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3960 [2026-10-21 15:59:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8501 [2026-10-21 16:00:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5428 [2026-10-21 16:00:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4498 [2026-10-21 16:00:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1731 [2026-10-21 16:00:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4638 [2026-10-21 16:00:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8784 [2026-10-21 16:00:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4828 [2026-10-21 16:00:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4275 [2026-10-21 16:00:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=3.4811 [2026-10-21 16:00:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4017 [2026-10-21 16:00:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9260 [2026-10-21 16:00:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1332 [2026-10-21 16:00:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0113 [2026-10-21 16:00:57][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,610 | MFU: 1.8% [2026-10-21 16:01:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4361 [2026-10-21 16:01:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6255 [2026-10-21 16:01:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5740 [2026-10-21 16:01:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5135 [2026-10-21 16:01:01][INFO][training.pretrain_engine] [Ph2|Step 231] loss=5.1957 | ppl=180.49 | aux=0.02507 | lr=2.99e-04 | gnorm=0.586 | tok/s=29,390 | MFU=7.1% | step=142710ms | tokens=1.044B | mem=4.2/32.7GB | cost=$18.94/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 03:54 (131.9h) | Phase3: SKIPPED | SFT: Oct 27 08:38 (136.6h) | DPO (Final): Oct 27 09:12 (137.2h) [2026-10-21 16:01:01][INFO][training.pretrain_engine] Starting global step 232 (accumulating 32 micro-batches)... [2026-10-21 16:01:01][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 16:01:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0872 [2026-10-21 16:01:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5891 [2026-10-21 16:01:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7326 [2026-10-21 16:01:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5994 [2026-10-21 16:01:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6826 [2026-10-21 16:01:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0710 [2026-10-21 16:01:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3704 [2026-10-21 16:01:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6400 [2026-10-21 16:01:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4277 [2026-10-21 16:01:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1918 [2026-10-21 16:01:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1622 [2026-10-21 16:01:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2371 [2026-10-21 16:02:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3380 [2026-10-21 16:02:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8932 [2026-10-21 16:02:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3946 [2026-10-21 16:02:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7410 [2026-10-21 16:02:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8221 [2026-10-21 16:02:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2010 [2026-10-21 16:02:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6263 [2026-10-21 16:02:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3646 [2026-10-21 16:02:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6327 [2026-10-21 16:02:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5790 [2026-10-21 16:02:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3214 [2026-10-21 16:02:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1896 [2026-10-21 16:03:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2933 [2026-10-21 16:03:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8254 [2026-10-21 16:03:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5691 [2026-10-21 16:03:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0513 [2026-10-21 16:03:19][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,623 | MFU: 1.8% [2026-10-21 16:03:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1487 [2026-10-21 16:03:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0839 [2026-10-21 16:03:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6004 [2026-10-21 16:03:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1806 [2026-10-21 16:03:24][INFO][training.pretrain_engine] [Ph2|Step 232] loss=5.1405 | ppl=170.80 | aux=0.02507 | lr=2.99e-04 | gnorm=0.586 | tok/s=29,390 | MFU=7.1% | step=142712ms | tokens=1.045B | mem=4.2/32.7GB | cost=$19.02/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 03:56 (131.9h) | Phase3: SKIPPED | SFT: Oct 27 08:40 (136.6h) | DPO (Final): Oct 27 09:14 (137.2h) [2026-10-21 16:03:24][INFO][training.pretrain_engine] Starting global step 233 (accumulating 32 micro-batches)... [2026-10-21 16:03:24][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 16:03:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9195 [2026-10-21 16:03:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9903 [2026-10-21 16:03:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1570 [2026-10-21 16:03:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4508 [2026-10-21 16:04:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3030 [2026-10-21 16:04:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3472 [2026-10-21 16:04:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2559 [2026-10-21 16:04:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9921 [2026-10-21 16:04:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2512 [2026-10-21 16:04:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3873 [2026-10-21 16:04:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4753 [2026-10-21 16:04:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0993 [2026-10-21 16:04:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4591 [2026-10-21 16:04:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4652 [2026-10-21 16:04:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4681 [2026-10-21 16:04:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2350 [2026-10-21 16:04:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2858 [2026-10-21 16:04:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4523 [2026-10-21 16:04:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2084 [2026-10-21 16:04:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5188 [2026-10-21 16:05:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8537 [2026-10-21 16:05:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6311 [2026-10-21 16:05:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8648 [2026-10-21 16:05:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7890 [2026-10-21 16:05:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2210 [2026-10-21 16:05:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1068 [2026-10-21 16:05:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9523 [2026-10-21 16:05:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8425 [2026-10-21 16:05:42][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,608 | MFU: 1.8% [2026-10-21 16:05:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9273 [2026-10-21 16:05:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0121 [2026-10-21 16:05:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9277 [2026-10-21 16:05:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8448 [2026-10-21 16:05:47][INFO][training.pretrain_engine] [Ph2|Step 233] loss=5.2253 | ppl=185.91 | aux=0.02506 | lr=2.99e-04 | gnorm=0.504 | tok/s=29,391 | MFU=7.1% | step=142707ms | tokens=1.046B | mem=4.2/32.7GB | cost=$19.10/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 03:58 (131.9h) | Phase3: SKIPPED | SFT: Oct 27 08:41 (136.6h) | DPO (Final): Oct 27 09:15 (137.2h) [2026-10-21 16:05:47][INFO][training.pretrain_engine] Starting global step 234 (accumulating 32 micro-batches)... [2026-10-21 16:05:47][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 16:06:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.5076 [2026-10-21 16:06:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5194 [2026-10-21 16:06:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6542 [2026-10-21 16:06:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9827 [2026-10-21 16:06:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2561 [2026-10-21 16:06:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9471 [2026-10-21 16:06:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5101 [2026-10-21 16:06:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8030 [2026-10-21 16:06:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8194 [2026-10-21 16:06:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9828 [2026-10-21 16:06:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9531 [2026-10-21 16:06:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3604 [2026-10-21 16:06:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4917 [2026-10-21 16:06:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3396 [2026-10-21 16:06:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5702 [2026-10-21 16:06:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0035 [2026-10-21 16:07:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5333 [2026-10-21 16:07:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6306 [2026-10-21 16:07:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9274 [2026-10-21 16:07:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7404 [2026-10-21 16:07:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8220 [2026-10-21 16:07:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4217 [2026-10-21 16:07:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0414 [2026-10-21 16:07:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9764 [2026-10-21 16:07:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0593 [2026-10-21 16:07:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0275 [2026-10-21 16:07:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1657 [2026-10-21 16:07:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1214 [2026-10-21 16:08:05][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,617 | MFU: 1.8% [2026-10-21 16:08:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9160 [2026-10-21 16:08:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7746 [2026-10-21 16:08:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2762 [2026-10-21 16:08:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3033 [2026-10-21 16:08:09][INFO][training.pretrain_engine] [Ph2|Step 234] loss=5.1563 | ppl=173.51 | aux=0.02507 | lr=2.99e-04 | gnorm=0.586 | tok/s=29,393 | MFU=7.1% | step=142699ms | tokens=1.048B | mem=4.2/32.7GB | cost=$19.18/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 03:59 (131.9h) | Phase3: SKIPPED | SFT: Oct 27 08:43 (136.6h) | DPO (Final): Oct 27 09:17 (137.2h) [2026-10-21 16:08:12][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step234 | Tokens: 1.048B | Spent: $19.18 | Remaining: $980.82 [2026-10-21 16:08:12][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step234 [2026-10-21 16:08:13][INFO][training.pretrain_engine] Starting global step 235 (accumulating 32 micro-batches)... [2026-10-21 16:08:13][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 16:08:13][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 16:08:13][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 16:08:13][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 16:08:13][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step234 to Phase-Technologies/forge-500m-pretrain (path: phase2_step234)... [2026-10-21 16:08:13][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 16:08:24][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 16:08:24][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step234 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 16:08:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8069 [2026-10-21 16:08:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9740 [2026-10-21 16:08:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6195 [2026-10-21 16:08:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8838 [2026-10-21 16:08:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3700 [2026-10-21 16:08:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6107 [2026-10-21 16:08:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6177 [2026-10-21 16:08:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3129 [2026-10-21 16:09:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5699 [2026-10-21 16:09:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0979 [2026-10-21 16:09:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6588 [2026-10-21 16:09:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2076 [2026-10-21 16:09:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1904 [2026-10-21 16:09:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7407 [2026-10-21 16:09:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8822 [2026-10-21 16:09:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6067 [2026-10-21 16:09:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0738 [2026-10-21 16:09:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0683 [2026-10-21 16:09:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3901 [2026-10-21 16:09:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3746 [2026-10-21 16:09:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3309 [2026-10-21 16:09:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3209 [2026-10-21 16:09:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6827 [2026-10-21 16:09:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0309 [2026-10-21 16:10:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0710 [2026-10-21 16:10:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3040 [2026-10-21 16:10:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4262 [2026-10-21 16:10:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9482 [2026-10-21 16:10:30][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,606 | MFU: 1.8% [2026-10-21 16:10:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6094 [2026-10-21 16:10:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7447 [2026-10-21 16:10:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6170 [2026-10-21 16:10:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5203 [2026-10-21 16:10:35][INFO][training.pretrain_engine] [Ph2|Step 235] loss=5.2830 | ppl=196.97 | aux=0.02507 | lr=2.99e-04 | gnorm=0.555 | tok/s=29,395 | MFU=7.1% | step=142690ms | tokens=1.049B | mem=4.2/32.7GB | cost=$19.26/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 04:01 (131.8h) | Phase3: SKIPPED | SFT: Oct 27 08:44 (136.6h) | DPO (Final): Oct 27 09:18 (137.1h) [2026-10-21 16:10:35][INFO][training.pretrain_engine] Starting global step 236 (accumulating 32 micro-batches)... [2026-10-21 16:10:35][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 16:10:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0226 [2026-10-21 16:10:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0830 [2026-10-21 16:10:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1998 [2026-10-21 16:10:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0663 [2026-10-21 16:11:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4818 [2026-10-21 16:11:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8189 [2026-10-21 16:11:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1684 [2026-10-21 16:11:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3194 [2026-10-21 16:11:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3881 [2026-10-21 16:11:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0326 [2026-10-21 16:11:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3116 [2026-10-21 16:11:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7693 [2026-10-21 16:11:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4152 [2026-10-21 16:11:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=3.8444 [2026-10-21 16:11:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0155 [2026-10-21 16:11:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3624 [2026-10-21 16:12:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3241 [2026-10-21 16:12:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3186 [2026-10-21 16:12:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2349 [2026-10-21 16:12:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3415 [2026-10-21 16:12:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2888 [2026-10-21 16:12:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1879 [2026-10-21 16:12:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0807 [2026-10-21 16:12:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2438 [2026-10-21 16:12:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4092 [2026-10-21 16:12:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5072 [2026-10-21 16:12:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7031 [2026-10-21 16:12:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2746 [2026-10-21 16:12:53][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,615 | MFU: 1.8% [2026-10-21 16:12:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4721 [2026-10-21 16:12:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0024 [2026-10-21 16:12:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1838 [2026-10-21 16:12:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1055 [2026-10-21 16:12:58][INFO][training.pretrain_engine] [Ph2|Step 236] loss=5.1650 | ppl=175.04 | aux=0.02507 | lr=2.99e-04 | gnorm=0.551 | tok/s=29,392 | MFU=7.1% | step=142700ms | tokens=1.050B | mem=4.2/32.7GB | cost=$19.35/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 04:03 (131.8h) | Phase3: SKIPPED | SFT: Oct 27 08:46 (136.6h) | DPO (Final): Oct 27 09:20 (137.1h) [2026-10-21 16:12:58][INFO][training.pretrain_engine] Starting global step 237 (accumulating 32 micro-batches)... [2026-10-21 16:12:58][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 16:13:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9648 [2026-10-21 16:13:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2456 [2026-10-21 16:13:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8411 [2026-10-21 16:13:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1723 [2026-10-21 16:13:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9289 [2026-10-21 16:13:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7296 [2026-10-21 16:13:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8451 [2026-10-21 16:13:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1548 [2026-10-21 16:13:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8573 [2026-10-21 16:13:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4939 [2026-10-21 16:13:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1147 [2026-10-21 16:13:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2786 [2026-10-21 16:14:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8549 [2026-10-21 16:14:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6771 [2026-10-21 16:14:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7437 [2026-10-21 16:14:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8294 [2026-10-21 16:14:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4593 [2026-10-21 16:14:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2612 [2026-10-21 16:14:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1951 [2026-10-21 16:14:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5120 [2026-10-21 16:14:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6328 [2026-10-21 16:14:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6362 [2026-10-21 16:14:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7338 [2026-10-21 16:14:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1466 [2026-10-21 16:15:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2200 [2026-10-21 16:15:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0774 [2026-10-21 16:15:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2475 [2026-10-21 16:15:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1136 [2026-10-21 16:15:16][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,611 | MFU: 1.8% [2026-10-21 16:15:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7630 [2026-10-21 16:15:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8833 [2026-10-21 16:15:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8407 [2026-10-21 16:15:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9809 [2026-10-21 16:15:21][INFO][training.pretrain_engine] [Ph2|Step 237] loss=5.1946 | ppl=180.29 | aux=0.02506 | lr=2.99e-04 | gnorm=0.475 | tok/s=29,393 | MFU=7.1% | step=142696ms | tokens=1.051B | mem=4.2/32.7GB | cost=$19.43/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 04:04 (131.8h) | Phase3: SKIPPED | SFT: Oct 27 08:48 (136.6h) | DPO (Final): Oct 27 09:22 (137.1h) [2026-10-21 16:15:21][INFO][training.pretrain_engine] Starting global step 238 (accumulating 32 micro-batches)... [2026-10-21 16:15:21][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 16:15:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1905 [2026-10-21 16:15:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3556 [2026-10-21 16:15:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5674 [2026-10-21 16:15:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3652 [2026-10-21 16:15:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3014 [2026-10-21 16:15:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3501 [2026-10-21 16:15:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2437 [2026-10-21 16:15:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1482 [2026-10-21 16:16:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2321 [2026-10-21 16:16:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2321 [2026-10-21 16:16:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0765 [2026-10-21 16:16:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3748 [2026-10-21 16:16:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4894 [2026-10-21 16:16:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0589 [2026-10-21 16:16:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0448 [2026-10-21 16:16:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7773 [2026-10-21 16:16:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1399 [2026-10-21 16:16:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4031 [2026-10-21 16:16:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2805 [2026-10-21 16:16:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6497 [2026-10-21 16:17:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0838 [2026-10-21 16:17:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3144 [2026-10-21 16:17:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9847 [2026-10-21 16:17:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9202 [2026-10-21 16:17:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5497 [2026-10-21 16:17:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2487 [2026-10-21 16:17:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6723 [2026-10-21 16:17:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3173 [2026-10-21 16:17:38][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,621 | MFU: 1.8% [2026-10-21 16:17:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1776 [2026-10-21 16:17:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0917 [2026-10-21 16:17:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2484 [2026-10-21 16:17:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0041 [2026-10-21 16:17:43][INFO][training.pretrain_engine] [Ph2|Step 238] loss=5.1551 | ppl=173.31 | aux=0.02507 | lr=2.99e-04 | gnorm=0.523 | tok/s=29,393 | MFU=7.1% | step=142696ms | tokens=1.052B | mem=4.2/32.7GB | cost=$19.51/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 04:06 (131.8h) | Phase3: SKIPPED | SFT: Oct 27 08:50 (136.5h) | DPO (Final): Oct 27 09:24 (137.1h) [2026-10-21 16:17:43][INFO][training.pretrain_engine] Starting global step 239 (accumulating 32 micro-batches)... [2026-10-21 16:17:43][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 16:18:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2340 [2026-10-21 16:18:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0528 [2026-10-21 16:18:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5496 [2026-10-21 16:18:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3809 [2026-10-21 16:18:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8073 [2026-10-21 16:18:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7661 [2026-10-21 16:18:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5264 [2026-10-21 16:18:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4196 [2026-10-21 16:18:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3052 [2026-10-21 16:18:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6579 [2026-10-21 16:18:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2223 [2026-10-21 16:18:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9681 [2026-10-21 16:18:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2385 [2026-10-21 16:18:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0093 [2026-10-21 16:18:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6876 [2026-10-21 16:18:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9460 [2026-10-21 16:19:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8743 [2026-10-21 16:19:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3431 [2026-10-21 16:19:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0039 [2026-10-21 16:19:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7992 [2026-10-21 16:19:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0737 [2026-10-21 16:19:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2984 [2026-10-21 16:19:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2321 [2026-10-21 16:19:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4967 [2026-10-21 16:19:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3214 [2026-10-21 16:19:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5302 [2026-10-21 16:19:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4306 [2026-10-21 16:19:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0657 [2026-10-21 16:20:01][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,613 | MFU: 1.8% [2026-10-21 16:20:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5879 [2026-10-21 16:20:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1785 [2026-10-21 16:20:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4170 [2026-10-21 16:20:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3771 [2026-10-21 16:20:06][INFO][training.pretrain_engine] [Ph2|Step 239] loss=5.2374 | ppl=188.18 | aux=0.02507 | lr=2.99e-04 | gnorm=0.473 | tok/s=29,392 | MFU=7.1% | step=142701ms | tokens=1.053B | mem=4.2/32.7GB | cost=$19.59/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 04:08 (131.8h) | Phase3: SKIPPED | SFT: Oct 27 08:52 (136.5h) | DPO (Final): Oct 27 09:26 (137.1h) [2026-10-21 16:20:06][INFO][training.pretrain_engine] Starting global step 240 (accumulating 32 micro-batches)... [2026-10-21 16:20:06][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 16:20:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3835 [2026-10-21 16:20:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4016 [2026-10-21 16:20:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1905 [2026-10-21 16:20:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4777 [2026-10-21 16:20:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4594 [2026-10-21 16:20:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2219 [2026-10-21 16:20:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3931 [2026-10-21 16:20:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9157 [2026-10-21 16:20:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1076 [2026-10-21 16:21:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9140 [2026-10-21 16:21:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=3.6998 [2026-10-21 16:21:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2481 [2026-10-21 16:21:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3786 [2026-10-21 16:21:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5726 [2026-10-21 16:21:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2549 [2026-10-21 16:21:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5420 [2026-10-21 16:21:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5023 [2026-10-21 16:21:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1667 [2026-10-21 16:21:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9751 [2026-10-21 16:21:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1447 [2026-10-21 16:21:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8314 [2026-10-21 16:21:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0347 [2026-10-21 16:21:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7876 [2026-10-21 16:21:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0365 [2026-10-21 16:22:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9313 [2026-10-21 16:22:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6441 [2026-10-21 16:22:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0355 [2026-10-21 16:22:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0415 [2026-10-21 16:22:24][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,600 | MFU: 1.8% [2026-10-21 16:22:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8723 [2026-10-21 16:22:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8523 [2026-10-21 16:22:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9348 [2026-10-21 16:22:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2890 [2026-10-21 16:22:29][INFO][training.pretrain_engine] [Ph2|Step 240] loss=5.1380 | ppl=170.37 | aux=0.02507 | lr=2.99e-04 | gnorm=0.490 | tok/s=29,392 | MFU=7.1% | step=142702ms | tokens=1.054B | mem=4.2/32.7GB | cost=$19.67/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 04:10 (131.8h) | Phase3: SKIPPED | SFT: Oct 27 08:54 (136.5h) | DPO (Final): Oct 27 09:28 (137.1h) [2026-10-21 16:22:29][INFO][training.pretrain_engine] Starting global step 241 (accumulating 32 micro-batches)... [2026-10-21 16:22:29][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 16:22:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4119 [2026-10-21 16:22:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1282 [2026-10-21 16:22:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5915 [2026-10-21 16:22:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7299 [2026-10-21 16:23:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0834 [2026-10-21 16:23:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4876 [2026-10-21 16:23:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2687 [2026-10-21 16:23:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4321 [2026-10-21 16:23:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9981 [2026-10-21 16:23:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0971 [2026-10-21 16:23:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2986 [2026-10-21 16:23:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2772 [2026-10-21 16:23:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1810 [2026-10-21 16:23:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2688 [2026-10-21 16:23:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9988 [2026-10-21 16:23:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4885 [2026-10-21 16:23:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4422 [2026-10-21 16:23:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2321 [2026-10-21 16:23:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3610 [2026-10-21 16:23:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0121 [2026-10-21 16:24:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6417 [2026-10-21 16:24:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4230 [2026-10-21 16:24:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0216 [2026-10-21 16:24:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5345 [2026-10-21 16:24:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9069 [2026-10-21 16:24:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8743 [2026-10-21 16:24:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6531 [2026-10-21 16:24:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3627 [2026-10-21 16:24:47][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,599 | MFU: 1.8% [2026-10-21 16:24:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4484 [2026-10-21 16:24:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3836 [2026-10-21 16:24:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0375 [2026-10-21 16:24:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4322 [2026-10-21 16:24:52][INFO][training.pretrain_engine] [Ph2|Step 241] loss=5.2465 | ppl=189.91 | aux=0.02508 | lr=2.99e-04 | gnorm=0.586 | tok/s=29,392 | MFU=7.1% | step=142703ms | tokens=1.055B | mem=4.2/32.7GB | cost=$19.75/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 04:12 (131.8h) | Phase3: SKIPPED | SFT: Oct 27 08:55 (136.5h) | DPO (Final): Oct 27 09:29 (137.1h) [2026-10-21 16:24:52][INFO][training.pretrain_engine] Starting global step 242 (accumulating 32 micro-batches)... [2026-10-21 16:24:52][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 16:25:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4540 [2026-10-21 16:25:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2423 [2026-10-21 16:25:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1921 [2026-10-21 16:25:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1756 [2026-10-21 16:25:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4212 [2026-10-21 16:25:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7214 [2026-10-21 16:25:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0426 [2026-10-21 16:25:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2174 [2026-10-21 16:25:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2771 [2026-10-21 16:25:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1222 [2026-10-21 16:25:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3027 [2026-10-21 16:25:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8365 [2026-10-21 16:26:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8010 [2026-10-21 16:26:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1385 [2026-10-21 16:26:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0048 [2026-10-21 16:26:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9247 [2026-10-21 16:26:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4920 [2026-10-21 16:26:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3493 [2026-10-21 16:26:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4179 [2026-10-21 16:26:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3753 [2026-10-21 16:26:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6815 [2026-10-21 16:26:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3742 [2026-10-21 16:26:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8876 [2026-10-21 16:26:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9369 [2026-10-21 16:26:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6657 [2026-10-21 16:26:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9152 [2026-10-21 16:26:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9120 [2026-10-21 16:26:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9993 [2026-10-21 16:27:10][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,610 | MFU: 1.8% [2026-10-21 16:27:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3769 [2026-10-21 16:27:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3008 [2026-10-21 16:27:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5486 [2026-10-21 16:27:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4755 [2026-10-21 16:27:15][INFO][training.pretrain_engine] [Ph2|Step 242] loss=5.2424 | ppl=189.12 | aux=0.02507 | lr=2.99e-04 | gnorm=0.707 | tok/s=29,392 | MFU=7.1% | step=142702ms | tokens=1.056B | mem=4.2/32.7GB | cost=$19.84/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 04:14 (131.8h) | Phase3: SKIPPED | SFT: Oct 27 08:57 (136.5h) | DPO (Final): Oct 27 09:31 (137.1h) [2026-10-21 16:27:15][INFO][training.pretrain_engine] Starting global step 243 (accumulating 32 micro-batches)... [2026-10-21 16:27:15][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 16:27:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9964 [2026-10-21 16:27:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6796 [2026-10-21 16:27:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3179 [2026-10-21 16:27:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1128 [2026-10-21 16:27:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8930 [2026-10-21 16:27:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0911 [2026-10-21 16:27:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1859 [2026-10-21 16:27:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0150 [2026-10-21 16:28:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0270 [2026-10-21 16:28:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3128 [2026-10-21 16:28:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8981 [2026-10-21 16:28:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4464 [2026-10-21 16:28:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1289 [2026-10-21 16:28:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1578 [2026-10-21 16:28:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2467 [2026-10-21 16:28:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0525 [2026-10-21 16:28:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4126 [2026-10-21 16:28:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0526 [2026-10-21 16:28:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3952 [2026-10-21 16:28:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2982 [2026-10-21 16:29:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1463 [2026-10-21 16:29:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3746 [2026-10-21 16:29:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7899 [2026-10-21 16:29:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8386 [2026-10-21 16:29:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7861 [2026-10-21 16:29:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7443 [2026-10-21 16:29:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7482 [2026-10-21 16:29:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7340 [2026-10-21 16:29:32][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,623 | MFU: 1.8% [2026-10-21 16:29:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4951 [2026-10-21 16:29:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6680 [2026-10-21 16:29:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1849 [2026-10-21 16:29:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4914 [2026-10-21 16:29:38][INFO][training.pretrain_engine] [Ph2|Step 243] loss=5.2604 | ppl=192.57 | aux=0.02506 | lr=2.99e-04 | gnorm=0.918 | tok/s=29,391 | MFU=7.1% | step=142710ms | tokens=1.057B | mem=4.2/32.7GB | cost=$19.92/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 04:16 (131.8h) | Phase3: SKIPPED | SFT: Oct 27 08:59 (136.5h) | DPO (Final): Oct 27 09:33 (137.1h) [2026-10-21 16:29:41][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step243 | Tokens: 1.057B | Spent: $19.92 | Remaining: $980.08 [2026-10-21 16:29:41][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step243 [2026-10-21 16:29:41][INFO][training.pretrain_engine] Starting global step 244 (accumulating 32 micro-batches)... [2026-10-21 16:29:41][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 16:29:41][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 16:29:41][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 16:29:41][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 16:29:41][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step243 to Phase-Technologies/forge-500m-pretrain (path: phase2_step243)... [2026-10-21 16:29:41][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 16:29:52][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 16:29:52][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step243 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 16:29:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0795 [2026-10-21 16:29:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5188 [2026-10-21 16:29:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6709 [2026-10-21 16:29:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5787 [2026-10-21 16:30:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4388 [2026-10-21 16:30:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5105 [2026-10-21 16:30:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1894 [2026-10-21 16:30:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4288 [2026-10-21 16:30:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0407 [2026-10-21 16:30:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4537 [2026-10-21 16:30:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5575 [2026-10-21 16:30:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7028 [2026-10-21 16:30:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8784 [2026-10-21 16:30:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9042 [2026-10-21 16:30:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4105 [2026-10-21 16:30:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7105 [2026-10-21 16:31:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1327 [2026-10-21 16:31:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6254 [2026-10-21 16:31:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3790 [2026-10-21 16:31:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6613 [2026-10-21 16:31:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9768 [2026-10-21 16:31:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2349 [2026-10-21 16:31:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9964 [2026-10-21 16:31:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8485 [2026-10-21 16:31:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5429 [2026-10-21 16:31:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8287 [2026-10-21 16:31:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5651 [2026-10-21 16:31:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2041 [2026-10-21 16:31:59][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,610 | MFU: 1.8% [2026-10-21 16:32:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3193 [2026-10-21 16:32:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8940 [2026-10-21 16:32:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7644 [2026-10-21 16:32:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8141 [2026-10-21 16:32:03][INFO][training.pretrain_engine] [Ph2|Step 244] loss=5.2235 | ppl=185.58 | aux=0.02507 | lr=2.99e-04 | gnorm=1.297 | tok/s=29,390 | MFU=7.1% | step=142710ms | tokens=1.058B | mem=4.2/32.7GB | cost=$20.00/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 04:18 (131.8h) | Phase3: SKIPPED | SFT: Oct 27 09:01 (136.5h) | DPO (Final): Oct 27 09:35 (137.1h) [2026-10-21 16:32:03][INFO][training.pretrain_engine] Starting global step 245 (accumulating 32 micro-batches)... [2026-10-21 16:32:03][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 16:32:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4344 [2026-10-21 16:32:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2375 [2026-10-21 16:32:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3035 [2026-10-21 16:32:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2320 [2026-10-21 16:32:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9792 [2026-10-21 16:32:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9215 [2026-10-21 16:32:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5995 [2026-10-21 16:32:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6292 [2026-10-21 16:32:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0378 [2026-10-21 16:32:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0906 [2026-10-21 16:32:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9526 [2026-10-21 16:32:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4646 [2026-10-21 16:33:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3014 [2026-10-21 16:33:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2059 [2026-10-21 16:33:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2050 [2026-10-21 16:33:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2972 [2026-10-21 16:33:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8322 [2026-10-21 16:33:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2573 [2026-10-21 16:33:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4794 [2026-10-21 16:33:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0456 [2026-10-21 16:33:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9776 [2026-10-21 16:33:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2131 [2026-10-21 16:33:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1637 [2026-10-21 16:33:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3208 [2026-10-21 16:34:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5296 [2026-10-21 16:34:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3909 [2026-10-21 16:34:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6298 [2026-10-21 16:34:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4967 [2026-10-21 16:34:21][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,592 | MFU: 1.8% [2026-10-21 16:34:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6842 [2026-10-21 16:34:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1288 [2026-10-21 16:34:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1839 [2026-10-21 16:34:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1031 [2026-10-21 16:34:26][INFO][training.pretrain_engine] [Ph2|Step 245] loss=5.1462 | ppl=171.77 | aux=0.02506 | lr=2.99e-04 | gnorm=0.812 | tok/s=29,389 | MFU=7.1% | step=142715ms | tokens=1.059B | mem=4.2/32.7GB | cost=$20.08/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 04:20 (131.8h) | Phase3: SKIPPED | SFT: Oct 27 09:03 (136.5h) | DPO (Final): Oct 27 09:37 (137.1h) [2026-10-21 16:34:26][INFO][training.pretrain_engine] Starting global step 246 (accumulating 32 micro-batches)... [2026-10-21 16:34:26][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 16:34:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1159 [2026-10-21 16:34:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5698 [2026-10-21 16:34:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1696 [2026-10-21 16:34:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4201 [2026-10-21 16:35:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2625 [2026-10-21 16:35:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1113 [2026-10-21 16:35:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3092 [2026-10-21 16:35:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4160 [2026-10-21 16:35:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6038 [2026-10-21 16:35:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4400 [2026-10-21 16:35:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1268 [2026-10-21 16:35:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.3547 [2026-10-21 16:35:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6847 [2026-10-21 16:35:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6834 [2026-10-21 16:35:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7378 [2026-10-21 16:35:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6788 [2026-10-21 16:35:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6814 [2026-10-21 16:35:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4450 [2026-10-21 16:35:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1371 [2026-10-21 16:35:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4930 [2026-10-21 16:36:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4653 [2026-10-21 16:36:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3555 [2026-10-21 16:36:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4963 [2026-10-21 16:36:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2815 [2026-10-21 16:36:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6319 [2026-10-21 16:36:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2226 [2026-10-21 16:36:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3709 [2026-10-21 16:36:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3552 [2026-10-21 16:36:44][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,592 | MFU: 1.8% [2026-10-21 16:36:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0310 [2026-10-21 16:36:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4942 [2026-10-21 16:36:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6207 [2026-10-21 16:36:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3467 [2026-10-21 16:36:49][INFO][training.pretrain_engine] [Ph2|Step 246] loss=5.2427 | ppl=189.18 | aux=0.02506 | lr=2.99e-04 | gnorm=0.977 | tok/s=29,387 | MFU=7.1% | step=142725ms | tokens=1.060B | mem=4.2/32.7GB | cost=$20.16/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 04:22 (131.8h) | Phase3: SKIPPED | SFT: Oct 27 09:06 (136.5h) | DPO (Final): Oct 27 09:40 (137.1h) [2026-10-21 16:36:49][INFO][training.pretrain_engine] Starting global step 247 (accumulating 32 micro-batches)... [2026-10-21 16:36:49][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 16:37:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4459 [2026-10-21 16:37:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0575 [2026-10-21 16:37:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5977 [2026-10-21 16:37:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7020 [2026-10-21 16:37:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1864 [2026-10-21 16:37:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0646 [2026-10-21 16:37:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4224 [2026-10-21 16:37:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5472 [2026-10-21 16:37:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4301 [2026-10-21 16:37:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4092 [2026-10-21 16:37:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0413 [2026-10-21 16:37:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0184 [2026-10-21 16:38:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6158 [2026-10-21 16:38:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0665 [2026-10-21 16:38:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0056 [2026-10-21 16:38:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2517 [2026-10-21 16:38:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0904 [2026-10-21 16:38:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4967 [2026-10-21 16:38:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2692 [2026-10-21 16:38:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1398 [2026-10-21 16:38:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9885 [2026-10-21 16:38:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8961 [2026-10-21 16:38:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0629 [2026-10-21 16:38:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1726 [2026-10-21 16:38:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6449 [2026-10-21 16:38:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0112 [2026-10-21 16:38:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8901 [2026-10-21 16:38:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2637 [2026-10-21 16:39:07][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,595 | MFU: 1.8% [2026-10-21 16:39:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4228 [2026-10-21 16:39:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7251 [2026-10-21 16:39:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3165 [2026-10-21 16:39:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5957 [2026-10-21 16:39:13][INFO][training.pretrain_engine] [Ph2|Step 247] loss=5.1541 | ppl=173.14 | aux=0.02506 | lr=2.99e-04 | gnorm=0.754 | tok/s=29,386 | MFU=7.1% | step=142733ms | tokens=1.061B | mem=4.2/32.7GB | cost=$20.25/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 04:24 (131.8h) | Phase3: SKIPPED | SFT: Oct 27 09:08 (136.5h) | DPO (Final): Oct 27 09:42 (137.1h) [2026-10-21 16:39:13][INFO][training.pretrain_engine] Starting global step 248 (accumulating 32 micro-batches)... [2026-10-21 16:39:13][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 16:39:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2252 [2026-10-21 16:39:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3438 [2026-10-21 16:39:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3577 [2026-10-21 16:39:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2359 [2026-10-21 16:39:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3235 [2026-10-21 16:39:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2105 [2026-10-21 16:39:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9199 [2026-10-21 16:39:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0671 [2026-10-21 16:40:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2022 [2026-10-21 16:40:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2204 [2026-10-21 16:40:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7852 [2026-10-21 16:40:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3639 [2026-10-21 16:40:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6768 [2026-10-21 16:40:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9326 [2026-10-21 16:40:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9095 [2026-10-21 16:40:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4985 [2026-10-21 16:40:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1144 [2026-10-21 16:40:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1009 [2026-10-21 16:40:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0985 [2026-10-21 16:40:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2606 [2026-10-21 16:41:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0971 [2026-10-21 16:41:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1393 [2026-10-21 16:41:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3044 [2026-10-21 16:41:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2267 [2026-10-21 16:41:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1918 [2026-10-21 16:41:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3710 [2026-10-21 16:41:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0985 [2026-10-21 16:41:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2020 [2026-10-21 16:41:31][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,604 | MFU: 1.8% [2026-10-21 16:41:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5410 [2026-10-21 16:41:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2692 [2026-10-21 16:41:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1433 [2026-10-21 16:41:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3890 [2026-10-21 16:41:36][INFO][training.pretrain_engine] [Ph2|Step 248] loss=5.1817 | ppl=177.98 | aux=0.02506 | lr=2.99e-04 | gnorm=0.836 | tok/s=29,385 | MFU=7.1% | step=142735ms | tokens=1.062B | mem=4.2/32.7GB | cost=$20.33/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 04:26 (131.8h) | Phase3: SKIPPED | SFT: Oct 27 09:10 (136.5h) | DPO (Final): Oct 27 09:44 (137.0h) [2026-10-21 16:41:36][INFO][training.pretrain_engine] Starting global step 249 (accumulating 32 micro-batches)... [2026-10-21 16:41:36][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 16:41:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3178 [2026-10-21 16:41:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1861 [2026-10-21 16:41:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3408 [2026-10-21 16:41:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0725 [2026-10-21 16:42:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1261 [2026-10-21 16:42:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2537 [2026-10-21 16:42:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0305 [2026-10-21 16:42:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7587 [2026-10-21 16:42:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2012 [2026-10-21 16:42:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4352 [2026-10-21 16:42:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7106 [2026-10-21 16:42:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1665 [2026-10-21 16:42:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3497 [2026-10-21 16:42:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7276 [2026-10-21 16:42:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5303 [2026-10-21 16:42:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6063 [2026-10-21 16:43:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9352 [2026-10-21 16:43:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7979 [2026-10-21 16:43:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2047 [2026-10-21 16:43:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8210 [2026-10-21 16:43:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2293 [2026-10-21 16:43:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3733 [2026-10-21 16:43:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0385 [2026-10-21 16:43:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2815 [2026-10-21 16:43:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5526 [2026-10-21 16:43:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7653 [2026-10-21 16:43:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2538 [2026-10-21 16:43:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6228 [2026-10-21 16:43:54][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,593 | MFU: 1.8% [2026-10-21 16:43:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1106 [2026-10-21 16:43:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3737 [2026-10-21 16:43:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8325 [2026-10-21 16:43:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7715 [2026-10-21 16:43:59][INFO][training.pretrain_engine] [Ph2|Step 249] loss=5.1082 | ppl=165.38 | aux=0.02508 | lr=2.99e-04 | gnorm=0.758 | tok/s=29,384 | MFU=7.1% | step=142740ms | tokens=1.063B | mem=4.2/32.7GB | cost=$20.41/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 04:28 (131.7h) | Phase3: SKIPPED | SFT: Oct 27 09:12 (136.5h) | DPO (Final): Oct 27 09:46 (137.0h) [2026-10-21 16:43:59][INFO][training.pretrain_engine] Starting global step 250 (accumulating 32 micro-batches)... [2026-10-21 16:43:59][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 16:44:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8610 [2026-10-21 16:44:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8729 [2026-10-21 16:44:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8089 [2026-10-21 16:44:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0115 [2026-10-21 16:44:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9587 [2026-10-21 16:44:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1865 [2026-10-21 16:44:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8997 [2026-10-21 16:44:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1441 [2026-10-21 16:44:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1842 [2026-10-21 16:44:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1401 [2026-10-21 16:44:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9128 [2026-10-21 16:44:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1125 [2026-10-21 16:45:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4751 [2026-10-21 16:45:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1351 [2026-10-21 16:45:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4452 [2026-10-21 16:45:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1132 [2026-10-21 16:45:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3844 [2026-10-21 16:45:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8756 [2026-10-21 16:45:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1718 [2026-10-21 16:45:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1977 [2026-10-21 16:45:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4530 [2026-10-21 16:45:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1835 [2026-10-21 16:45:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2931 [2026-10-21 16:45:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4449 [2026-10-21 16:46:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0892 [2026-10-21 16:46:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5432 [2026-10-21 16:46:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5117 [2026-10-21 16:46:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2929 [2026-10-21 16:46:17][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,600 | MFU: 1.8% [2026-10-21 16:46:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5024 [2026-10-21 16:46:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8202 [2026-10-21 16:46:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3957 [2026-10-21 16:46:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6992 [2026-10-21 16:46:22][INFO][training.pretrain_engine] [Ph2|Step 250] loss=5.1336 | ppl=169.62 | aux=0.02507 | lr=2.99e-04 | gnorm=0.648 | tok/s=29,384 | MFU=7.1% | step=142740ms | tokens=1.064B | mem=4.2/32.7GB | cost=$20.49/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 04:30 (131.7h) | Phase3: SKIPPED | SFT: Oct 27 09:14 (136.5h) | DPO (Final): Oct 27 09:48 (137.0h) [2026-10-21 16:46:22][INFO][training.pretrain_engine] Starting global step 251 (accumulating 32 micro-batches)... [2026-10-21 16:46:22][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 16:46:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0207 [2026-10-21 16:46:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1669 [2026-10-21 16:46:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3962 [2026-10-21 16:46:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1022 [2026-10-21 16:46:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7710 [2026-10-21 16:46:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5924 [2026-10-21 16:46:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7361 [2026-10-21 16:46:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6764 [2026-10-21 16:47:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9100 [2026-10-21 16:47:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7769 [2026-10-21 16:47:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9250 [2026-10-21 16:47:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8311 [2026-10-21 16:47:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9966 [2026-10-21 16:47:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2467 [2026-10-21 16:47:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1721 [2026-10-21 16:47:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4577 [2026-10-21 16:47:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6888 [2026-10-21 16:47:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7808 [2026-10-21 16:47:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5113 [2026-10-21 16:47:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3296 [2026-10-21 16:48:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8590 [2026-10-21 16:48:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8684 [2026-10-21 16:48:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1645 [2026-10-21 16:48:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1065 [2026-10-21 16:48:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7443 [2026-10-21 16:48:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5041 [2026-10-21 16:48:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4656 [2026-10-21 16:48:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5133 [2026-10-21 16:48:40][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,622 | MFU: 1.8% [2026-10-21 16:48:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6623 [2026-10-21 16:48:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0899 [2026-10-21 16:48:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3209 [2026-10-21 16:48:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3677 [2026-10-21 16:48:45][INFO][training.pretrain_engine] [Ph2|Step 251] loss=5.1644 | ppl=174.94 | aux=0.02508 | lr=2.99e-04 | gnorm=0.664 | tok/s=29,384 | MFU=7.1% | step=142740ms | tokens=1.065B | mem=4.2/32.7GB | cost=$20.57/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 04:32 (131.7h) | Phase3: SKIPPED | SFT: Oct 27 09:16 (136.5h) | DPO (Final): Oct 27 09:50 (137.0h) [2026-10-21 16:48:45][INFO][training.pretrain_engine] Starting global step 252 (accumulating 32 micro-batches)... [2026-10-21 16:48:45][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 16:49:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9243 [2026-10-21 16:49:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6411 [2026-10-21 16:49:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9436 [2026-10-21 16:49:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0188 [2026-10-21 16:49:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5568 [2026-10-21 16:49:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4571 [2026-10-21 16:49:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3686 [2026-10-21 16:49:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3367 [2026-10-21 16:49:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0085 [2026-10-21 16:49:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0779 [2026-10-21 16:49:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=3.8624 [2026-10-21 16:49:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8147 [2026-10-21 16:49:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3741 [2026-10-21 16:49:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3109 [2026-10-21 16:49:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3346 [2026-10-21 16:49:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4734 [2026-10-21 16:50:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9223 [2026-10-21 16:50:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1422 [2026-10-21 16:50:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0731 [2026-10-21 16:50:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9758 [2026-10-21 16:50:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4822 [2026-10-21 16:50:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.1945 [2026-10-21 16:50:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3107 [2026-10-21 16:50:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9623 [2026-10-21 16:50:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6819 [2026-10-21 16:50:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6988 [2026-10-21 16:50:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5909 [2026-10-21 16:50:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0042 [2026-10-21 16:51:03][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,605 | MFU: 1.8% [2026-10-21 16:51:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3073 [2026-10-21 16:51:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2057 [2026-10-21 16:51:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1518 [2026-10-21 16:51:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3572 [2026-10-21 16:51:08][INFO][training.pretrain_engine] [Ph2|Step 252] loss=5.1874 | ppl=179.00 | aux=0.02507 | lr=2.99e-04 | gnorm=0.621 | tok/s=29,384 | MFU=7.1% | step=142742ms | tokens=1.066B | mem=4.2/32.7GB | cost=$20.66/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 04:34 (131.7h) | Phase3: SKIPPED | SFT: Oct 27 09:17 (136.4h) | DPO (Final): Oct 27 09:51 (137.0h) [2026-10-21 16:51:11][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step252 | Tokens: 1.066B | Spent: $20.66 | Remaining: $979.34 [2026-10-21 16:51:11][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step252 [2026-10-21 16:51:11][INFO][training.pretrain_engine] Starting global step 253 (accumulating 32 micro-batches)... [2026-10-21 16:51:11][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 16:51:12][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 16:51:12][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 16:51:12][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 16:51:12][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step252 to Phase-Technologies/forge-500m-pretrain (path: phase2_step252)... [2026-10-21 16:51:12][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 16:51:21][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 16:51:21][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step252 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 16:51:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3982 [2026-10-21 16:51:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2778 [2026-10-21 16:51:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2072 [2026-10-21 16:51:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2001 [2026-10-21 16:51:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2816 [2026-10-21 16:51:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7499 [2026-10-21 16:51:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0126 [2026-10-21 16:51:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9760 [2026-10-21 16:52:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2706 [2026-10-21 16:52:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5299 [2026-10-21 16:52:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2406 [2026-10-21 16:52:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2083 [2026-10-21 16:52:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0674 [2026-10-21 16:52:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3381 [2026-10-21 16:52:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8964 [2026-10-21 16:52:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3306 [2026-10-21 16:52:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6390 [2026-10-21 16:52:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5497 [2026-10-21 16:52:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6327 [2026-10-21 16:52:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8526 [2026-10-21 16:52:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0135 [2026-10-21 16:52:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4362 [2026-10-21 16:52:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4791 [2026-10-21 16:52:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.0305 [2026-10-21 16:53:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3697 [2026-10-21 16:53:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0994 [2026-10-21 16:53:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5379 [2026-10-21 16:53:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0529 [2026-10-21 16:53:29][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,616 | MFU: 1.8% [2026-10-21 16:53:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2867 [2026-10-21 16:53:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9698 [2026-10-21 16:53:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2231 [2026-10-21 16:53:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3811 [2026-10-21 16:53:34][INFO][training.pretrain_engine] [Ph2|Step 253] loss=5.2044 | ppl=182.08 | aux=0.02508 | lr=2.99e-04 | gnorm=0.680 | tok/s=29,385 | MFU=7.1% | step=142736ms | tokens=1.067B | mem=4.2/32.7GB | cost=$20.74/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 04:35 (131.7h) | Phase3: SKIPPED | SFT: Oct 27 09:19 (136.4h) | DPO (Final): Oct 27 09:53 (137.0h) [2026-10-21 16:53:34][INFO][training.pretrain_engine] Starting global step 254 (accumulating 32 micro-batches)... [2026-10-21 16:53:34][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 16:53:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2348 [2026-10-21 16:53:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6392 [2026-10-21 16:53:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5503 [2026-10-21 16:53:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6587 [2026-10-21 16:54:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2065 [2026-10-21 16:54:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0357 [2026-10-21 16:54:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0591 [2026-10-21 16:54:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3614 [2026-10-21 16:54:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0729 [2026-10-21 16:54:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4037 [2026-10-21 16:54:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6766 [2026-10-21 16:54:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9786 [2026-10-21 16:54:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1390 [2026-10-21 16:54:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3124 [2026-10-21 16:54:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7552 [2026-10-21 16:54:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0202 [2026-10-21 16:55:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1763 [2026-10-21 16:55:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3181 [2026-10-21 16:55:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3208 [2026-10-21 16:55:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2808 [2026-10-21 16:55:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8767 [2026-10-21 16:55:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9072 [2026-10-21 16:55:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7786 [2026-10-21 16:55:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4112 [2026-10-21 16:55:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0894 [2026-10-21 16:55:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0445 [2026-10-21 16:55:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1133 [2026-10-21 16:55:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2145 [2026-10-21 16:55:52][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,591 | MFU: 1.8% [2026-10-21 16:55:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1831 [2026-10-21 16:55:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2161 [2026-10-21 16:55:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2185 [2026-10-21 16:55:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9133 [2026-10-21 16:55:56][INFO][training.pretrain_engine] [Ph2|Step 254] loss=5.1566 | ppl=173.58 | aux=0.02507 | lr=2.99e-04 | gnorm=0.734 | tok/s=29,384 | MFU=7.1% | step=142741ms | tokens=1.068B | mem=4.2/32.7GB | cost=$20.82/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 04:37 (131.7h) | Phase3: SKIPPED | SFT: Oct 27 09:21 (136.4h) | DPO (Final): Oct 27 09:55 (137.0h) [2026-10-21 16:55:56][INFO][training.pretrain_engine] Starting global step 255 (accumulating 32 micro-batches)... [2026-10-21 16:55:56][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 16:56:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7635 [2026-10-21 16:56:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0715 [2026-10-21 16:56:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1903 [2026-10-21 16:56:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1455 [2026-10-21 16:56:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1623 [2026-10-21 16:56:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.9429 [2026-10-21 16:56:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1336 [2026-10-21 16:56:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2401 [2026-10-21 16:56:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2845 [2026-10-21 16:56:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9056 [2026-10-21 16:56:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1261 [2026-10-21 16:56:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3088 [2026-10-21 16:57:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2457 [2026-10-21 16:57:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0764 [2026-10-21 16:57:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9197 [2026-10-21 16:57:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0317 [2026-10-21 16:57:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2341 [2026-10-21 16:57:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9963 [2026-10-21 16:57:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3091 [2026-10-21 16:57:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2907 [2026-10-21 16:57:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8152 [2026-10-21 16:57:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1461 [2026-10-21 16:57:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9412 [2026-10-21 16:57:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9968 [2026-10-21 16:58:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7599 [2026-10-21 16:58:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2474 [2026-10-21 16:58:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9257 [2026-10-21 16:58:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1561 [2026-10-21 16:58:14][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,626 | MFU: 1.8% [2026-10-21 16:58:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2724 [2026-10-21 16:58:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5825 [2026-10-21 16:58:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9626 [2026-10-21 16:58:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9529 [2026-10-21 16:58:19][INFO][training.pretrain_engine] [Ph2|Step 255] loss=5.1442 | ppl=171.44 | aux=0.02506 | lr=2.99e-04 | gnorm=0.637 | tok/s=29,383 | MFU=7.1% | step=142745ms | tokens=1.070B | mem=4.2/32.7GB | cost=$20.90/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 04:39 (131.7h) | Phase3: SKIPPED | SFT: Oct 27 09:23 (136.4h) | DPO (Final): Oct 27 09:57 (137.0h) [2026-10-21 16:58:19][INFO][training.pretrain_engine] Starting global step 256 (accumulating 32 micro-batches)... [2026-10-21 16:58:19][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 16:58:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5365 [2026-10-21 16:58:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4598 [2026-10-21 16:58:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6731 [2026-10-21 16:58:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5489 [2026-10-21 16:58:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7726 [2026-10-21 16:58:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0080 [2026-10-21 16:58:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9290 [2026-10-21 16:58:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0847 [2026-10-21 16:59:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5769 [2026-10-21 16:59:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7220 [2026-10-21 16:59:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0638 [2026-10-21 16:59:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7418 [2026-10-21 16:59:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3094 [2026-10-21 16:59:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2161 [2026-10-21 16:59:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8118 [2026-10-21 16:59:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1374 [2026-10-21 16:59:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2753 [2026-10-21 16:59:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5527 [2026-10-21 16:59:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5234 [2026-10-21 16:59:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3887 [2026-10-21 17:00:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9574 [2026-10-21 17:00:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4269 [2026-10-21 17:00:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5171 [2026-10-21 17:00:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5220 [2026-10-21 17:00:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1690 [2026-10-21 17:00:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3299 [2026-10-21 17:00:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9771 [2026-10-21 17:00:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3297 [2026-10-21 17:00:37][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,604 | MFU: 1.8% [2026-10-21 17:00:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7616 [2026-10-21 17:00:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9168 [2026-10-21 17:00:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2515 [2026-10-21 17:00:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9979 [2026-10-21 17:00:41][INFO][training.pretrain_engine] [Ph2|Step 256] loss=5.2178 | ppl=184.53 | aux=0.02506 | lr=2.99e-04 | gnorm=0.566 | tok/s=29,383 | MFU=7.1% | step=142745ms | tokens=1.071B | mem=4.2/32.7GB | cost=$20.98/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 04:41 (131.7h) | Phase3: SKIPPED | SFT: Oct 27 09:25 (136.4h) | DPO (Final): Oct 27 09:59 (137.0h) [2026-10-21 17:00:41][INFO][training.pretrain_engine] Starting global step 257 (accumulating 32 micro-batches)... [2026-10-21 17:00:41][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 17:00:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3378 [2026-10-21 17:00:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2443 [2026-10-21 17:00:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4587 [2026-10-21 17:00:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3535 [2026-10-21 17:01:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3694 [2026-10-21 17:01:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2831 [2026-10-21 17:01:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7429 [2026-10-21 17:01:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7192 [2026-10-21 17:01:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3886 [2026-10-21 17:01:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4816 [2026-10-21 17:01:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4776 [2026-10-21 17:01:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3681 [2026-10-21 17:01:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8549 [2026-10-21 17:01:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8655 [2026-10-21 17:01:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3697 [2026-10-21 17:01:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3167 [2026-10-21 17:02:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1220 [2026-10-21 17:02:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2128 [2026-10-21 17:02:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9936 [2026-10-21 17:02:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1922 [2026-10-21 17:02:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5120 [2026-10-21 17:02:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3666 [2026-10-21 17:02:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3154 [2026-10-21 17:02:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4603 [2026-10-21 17:02:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9983 [2026-10-21 17:02:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8670 [2026-10-21 17:02:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3203 [2026-10-21 17:02:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8715 [2026-10-21 17:02:59][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,611 | MFU: 1.8% [2026-10-21 17:03:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8334 [2026-10-21 17:03:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8553 [2026-10-21 17:03:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3410 [2026-10-21 17:03:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7033 [2026-10-21 17:03:04][INFO][training.pretrain_engine] [Ph2|Step 257] loss=5.2088 | ppl=182.87 | aux=0.02507 | lr=2.99e-04 | gnorm=0.562 | tok/s=29,381 | MFU=7.1% | step=142754ms | tokens=1.072B | mem=4.2/32.7GB | cost=$21.07/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 04:43 (131.7h) | Phase3: SKIPPED | SFT: Oct 27 09:27 (136.4h) | DPO (Final): Oct 27 10:01 (137.0h) [2026-10-21 17:03:04][INFO][training.pretrain_engine] Starting global step 258 (accumulating 32 micro-batches)... [2026-10-21 17:03:04][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 17:03:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1342 [2026-10-21 17:03:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2955 [2026-10-21 17:03:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3401 [2026-10-21 17:03:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1799 [2026-10-21 17:03:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8387 [2026-10-21 17:03:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3657 [2026-10-21 17:03:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8070 [2026-10-21 17:03:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9080 [2026-10-21 17:03:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9846 [2026-10-21 17:03:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8474 [2026-10-21 17:03:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7368 [2026-10-21 17:03:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9882 [2026-10-21 17:04:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6218 [2026-10-21 17:04:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3479 [2026-10-21 17:04:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1420 [2026-10-21 17:04:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2971 [2026-10-21 17:04:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4250 [2026-10-21 17:04:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9189 [2026-10-21 17:04:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8888 [2026-10-21 17:04:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3118 [2026-10-21 17:04:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2949 [2026-10-21 17:04:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3524 [2026-10-21 17:04:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8465 [2026-10-21 17:04:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2033 [2026-10-21 17:05:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9055 [2026-10-21 17:05:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1656 [2026-10-21 17:05:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0760 [2026-10-21 17:05:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4009 [2026-10-21 17:05:22][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,608 | MFU: 1.8% [2026-10-21 17:05:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9563 [2026-10-21 17:05:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7141 [2026-10-21 17:05:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4623 [2026-10-21 17:05:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1188 [2026-10-21 17:05:27][INFO][training.pretrain_engine] [Ph2|Step 258] loss=5.1955 | ppl=180.45 | aux=0.02507 | lr=2.99e-04 | gnorm=0.547 | tok/s=29,381 | MFU=7.1% | step=142757ms | tokens=1.073B | mem=4.2/32.7GB | cost=$21.15/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 04:45 (131.7h) | Phase3: SKIPPED | SFT: Oct 27 09:29 (136.4h) | DPO (Final): Oct 27 10:03 (137.0h) [2026-10-21 17:05:27][INFO][training.pretrain_engine] Starting global step 259 (accumulating 32 micro-batches)... [2026-10-21 17:05:27][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 17:05:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1980 [2026-10-21 17:05:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9050 [2026-10-21 17:05:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8204 [2026-10-21 17:05:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1092 [2026-10-21 17:06:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1450 [2026-10-21 17:06:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5544 [2026-10-21 17:06:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6330 [2026-10-21 17:06:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7109 [2026-10-21 17:06:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2207 [2026-10-21 17:06:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3034 [2026-10-21 17:06:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5288 [2026-10-21 17:06:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3622 [2026-10-21 17:06:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8859 [2026-10-21 17:06:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0673 [2026-10-21 17:06:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2448 [2026-10-21 17:06:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1775 [2026-10-21 17:06:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1275 [2026-10-21 17:06:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2809 [2026-10-21 17:06:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2021 [2026-10-21 17:06:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0904 [2026-10-21 17:07:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9834 [2026-10-21 17:07:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1208 [2026-10-21 17:07:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0408 [2026-10-21 17:07:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3452 [2026-10-21 17:07:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5111 [2026-10-21 17:07:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9713 [2026-10-21 17:07:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6164 [2026-10-21 17:07:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4302 [2026-10-21 17:07:44][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,624 | MFU: 1.8% [2026-10-21 17:07:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3668 [2026-10-21 17:07:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1398 [2026-10-21 17:07:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6417 [2026-10-21 17:07:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8588 [2026-10-21 17:07:49][INFO][training.pretrain_engine] [Ph2|Step 259] loss=5.1810 | ppl=177.87 | aux=0.02506 | lr=2.99e-04 | gnorm=0.512 | tok/s=29,380 | MFU=7.1% | step=142762ms | tokens=1.074B | mem=4.2/32.7GB | cost=$21.23/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 04:47 (131.7h) | Phase3: SKIPPED | SFT: Oct 27 09:31 (136.4h) | DPO (Final): Oct 27 10:05 (137.0h) [2026-10-21 17:07:49][INFO][training.pretrain_engine] Starting global step 260 (accumulating 32 micro-batches)... [2026-10-21 17:07:49][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 17:08:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3196 [2026-10-21 17:08:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2194 [2026-10-21 17:08:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4122 [2026-10-21 17:08:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5589 [2026-10-21 17:08:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3143 [2026-10-21 17:08:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0460 [2026-10-21 17:08:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1725 [2026-10-21 17:08:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3691 [2026-10-21 17:08:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4448 [2026-10-21 17:08:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1588 [2026-10-21 17:08:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3608 [2026-10-21 17:08:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3166 [2026-10-21 17:09:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7863 [2026-10-21 17:09:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1254 [2026-10-21 17:09:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5147 [2026-10-21 17:09:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9303 [2026-10-21 17:09:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2326 [2026-10-21 17:09:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1275 [2026-10-21 17:09:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9780 [2026-10-21 17:09:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2087 [2026-10-21 17:09:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0014 [2026-10-21 17:09:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1961 [2026-10-21 17:09:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9027 [2026-10-21 17:09:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9074 [2026-10-21 17:09:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6442 [2026-10-21 17:09:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6705 [2026-10-21 17:09:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8015 [2026-10-21 17:09:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8356 [2026-10-21 17:10:07][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,616 | MFU: 1.8% [2026-10-21 17:10:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0845 [2026-10-21 17:10:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8983 [2026-10-21 17:10:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4564 [2026-10-21 17:10:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0900 [2026-10-21 17:10:12][INFO][training.pretrain_engine] [Ph2|Step 260] loss=5.1773 | ppl=177.20 | aux=0.02506 | lr=2.99e-04 | gnorm=0.488 | tok/s=29,380 | MFU=7.1% | step=142759ms | tokens=1.075B | mem=4.2/32.7GB | cost=$21.31/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 04:49 (131.7h) | Phase3: SKIPPED | SFT: Oct 27 09:33 (136.4h) | DPO (Final): Oct 27 10:07 (137.0h) [2026-10-21 17:10:12][INFO][training.pretrain_engine] Starting global step 261 (accumulating 32 micro-batches)... [2026-10-21 17:10:12][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 17:10:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4856 [2026-10-21 17:10:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9756 [2026-10-21 17:10:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7580 [2026-10-21 17:10:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8684 [2026-10-21 17:10:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2368 [2026-10-21 17:10:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.9332 [2026-10-21 17:10:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1306 [2026-10-21 17:10:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1295 [2026-10-21 17:11:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9263 [2026-10-21 17:11:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9374 [2026-10-21 17:11:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7343 [2026-10-21 17:11:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7447 [2026-10-21 17:11:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.1281 [2026-10-21 17:11:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6201 [2026-10-21 17:11:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7222 [2026-10-21 17:11:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4346 [2026-10-21 17:11:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6816 [2026-10-21 17:11:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5221 [2026-10-21 17:11:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4484 [2026-10-21 17:11:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2891 [2026-10-21 17:11:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2531 [2026-10-21 17:11:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8104 [2026-10-21 17:11:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7500 [2026-10-21 17:11:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5933 [2026-10-21 17:12:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7880 [2026-10-21 17:12:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8267 [2026-10-21 17:12:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3602 [2026-10-21 17:12:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7303 [2026-10-21 17:12:29][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,623 | MFU: 1.8% [2026-10-21 17:12:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9468 [2026-10-21 17:12:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8323 [2026-10-21 17:12:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1945 [2026-10-21 17:12:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.0292 [2026-10-21 17:12:34][INFO][training.pretrain_engine] [Ph2|Step 261] loss=5.1774 | ppl=177.22 | aux=0.02506 | lr=2.99e-04 | gnorm=0.461 | tok/s=29,382 | MFU=7.1% | step=142750ms | tokens=1.076B | mem=4.2/32.7GB | cost=$21.39/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 04:50 (131.6h) | Phase3: SKIPPED | SFT: Oct 27 09:34 (136.4h) | DPO (Final): Oct 27 10:08 (136.9h) [2026-10-21 17:12:37][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step261 | Tokens: 1.076B | Spent: $21.39 | Remaining: $978.61 [2026-10-21 17:12:37][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step261 [2026-10-21 17:12:37][INFO][training.pretrain_engine] Starting global step 262 (accumulating 32 micro-batches)... [2026-10-21 17:12:37][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 17:12:37][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 17:12:37][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 17:12:38][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 17:12:38][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step261 to Phase-Technologies/forge-500m-pretrain (path: phase2_step261)... [2026-10-21 17:12:38][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 17:12:50][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 17:12:50][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step261 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 17:12:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0178 [2026-10-21 17:12:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1930 [2026-10-21 17:12:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0426 [2026-10-21 17:12:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8156 [2026-10-21 17:13:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9692 [2026-10-21 17:13:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3383 [2026-10-21 17:13:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0626 [2026-10-21 17:13:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1835 [2026-10-21 17:13:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4284 [2026-10-21 17:13:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8524 [2026-10-21 17:13:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4273 [2026-10-21 17:13:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8188 [2026-10-21 17:13:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0664 [2026-10-21 17:13:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9603 [2026-10-21 17:13:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9638 [2026-10-21 17:13:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0256 [2026-10-21 17:14:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7708 [2026-10-21 17:14:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7541 [2026-10-21 17:14:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=3.9719 [2026-10-21 17:14:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9653 [2026-10-21 17:14:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1297 [2026-10-21 17:14:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1740 [2026-10-21 17:14:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0266 [2026-10-21 17:14:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2149 [2026-10-21 17:14:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2149 [2026-10-21 17:14:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8124 [2026-10-21 17:14:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1579 [2026-10-21 17:14:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0056 [2026-10-21 17:14:55][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,601 | MFU: 1.8% [2026-10-21 17:15:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4290 [2026-10-21 17:15:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0322 [2026-10-21 17:15:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2633 [2026-10-21 17:15:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6977 [2026-10-21 17:15:00][INFO][training.pretrain_engine] [Ph2|Step 262] loss=5.2214 | ppl=185.19 | aux=0.02508 | lr=2.99e-04 | gnorm=0.451 | tok/s=29,383 | MFU=7.1% | step=142746ms | tokens=1.077B | mem=4.2/32.7GB | cost=$21.47/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 04:52 (131.6h) | Phase3: SKIPPED | SFT: Oct 27 09:36 (136.4h) | DPO (Final): Oct 27 10:10 (136.9h) [2026-10-21 17:15:00][INFO][training.pretrain_engine] Starting global step 263 (accumulating 32 micro-batches)... [2026-10-21 17:15:00][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 17:15:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8076 [2026-10-21 17:15:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7473 [2026-10-21 17:15:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9474 [2026-10-21 17:15:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7355 [2026-10-21 17:15:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8806 [2026-10-21 17:15:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5296 [2026-10-21 17:15:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9638 [2026-10-21 17:15:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1661 [2026-10-21 17:15:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6927 [2026-10-21 17:15:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2248 [2026-10-21 17:15:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4064 [2026-10-21 17:15:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0762 [2026-10-21 17:16:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1742 [2026-10-21 17:16:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1057 [2026-10-21 17:16:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0906 [2026-10-21 17:16:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1657 [2026-10-21 17:16:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3226 [2026-10-21 17:16:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9186 [2026-10-21 17:16:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3146 [2026-10-21 17:16:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0918 [2026-10-21 17:16:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1474 [2026-10-21 17:16:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0353 [2026-10-21 17:16:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3403 [2026-10-21 17:16:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6619 [2026-10-21 17:17:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7474 [2026-10-21 17:17:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0993 [2026-10-21 17:17:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1223 [2026-10-21 17:17:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1692 [2026-10-21 17:17:17][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,617 | MFU: 1.8% [2026-10-21 17:17:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3442 [2026-10-21 17:17:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4713 [2026-10-21 17:17:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2982 [2026-10-21 17:17:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3523 [2026-10-21 17:17:22][INFO][training.pretrain_engine] [Ph2|Step 263] loss=5.1691 | ppl=175.76 | aux=0.02506 | lr=2.99e-04 | gnorm=0.469 | tok/s=29,382 | MFU=7.1% | step=142749ms | tokens=1.078B | mem=4.2/32.7GB | cost=$21.56/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 04:54 (131.6h) | Phase3: SKIPPED | SFT: Oct 27 09:38 (136.3h) | DPO (Final): Oct 27 10:12 (136.9h) [2026-10-21 17:17:22][INFO][training.pretrain_engine] Starting global step 264 (accumulating 32 micro-batches)... [2026-10-21 17:17:22][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 17:17:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2029 [2026-10-21 17:17:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4366 [2026-10-21 17:17:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2600 [2026-10-21 17:17:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2337 [2026-10-21 17:17:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2540 [2026-10-21 17:17:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1609 [2026-10-21 17:17:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3055 [2026-10-21 17:17:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6072 [2026-10-21 17:18:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2052 [2026-10-21 17:18:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2069 [2026-10-21 17:18:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3229 [2026-10-21 17:18:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0008 [2026-10-21 17:18:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6432 [2026-10-21 17:18:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6350 [2026-10-21 17:18:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6365 [2026-10-21 17:18:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7078 [2026-10-21 17:18:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3636 [2026-10-21 17:18:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4382 [2026-10-21 17:18:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5273 [2026-10-21 17:18:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3555 [2026-10-21 17:19:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4412 [2026-10-21 17:19:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7456 [2026-10-21 17:19:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9429 [2026-10-21 17:19:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7427 [2026-10-21 17:19:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6324 [2026-10-21 17:19:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2712 [2026-10-21 17:19:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7839 [2026-10-21 17:19:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5291 [2026-10-21 17:19:40][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,603 | MFU: 1.8% [2026-10-21 17:19:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=3.8859 [2026-10-21 17:19:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7833 [2026-10-21 17:19:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9041 [2026-10-21 17:19:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1951 [2026-10-21 17:19:45][INFO][training.pretrain_engine] [Ph2|Step 264] loss=5.1273 | ppl=168.55 | aux=0.02507 | lr=2.99e-04 | gnorm=0.490 | tok/s=29,382 | MFU=7.1% | step=142753ms | tokens=1.079B | mem=4.2/32.7GB | cost=$21.64/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 04:56 (131.6h) | Phase3: SKIPPED | SFT: Oct 27 09:40 (136.3h) | DPO (Final): Oct 27 10:14 (136.9h) [2026-10-21 17:19:45][INFO][training.pretrain_engine] Starting global step 265 (accumulating 32 micro-batches)... [2026-10-21 17:19:45][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 17:20:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1433 [2026-10-21 17:20:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1016 [2026-10-21 17:20:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3676 [2026-10-21 17:20:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8183 [2026-10-21 17:20:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0288 [2026-10-21 17:20:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2284 [2026-10-21 17:20:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5945 [2026-10-21 17:20:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8467 [2026-10-21 17:20:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1944 [2026-10-21 17:20:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1386 [2026-10-21 17:20:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3937 [2026-10-21 17:20:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7151 [2026-10-21 17:20:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5129 [2026-10-21 17:20:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1855 [2026-10-21 17:20:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8760 [2026-10-21 17:20:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6030 [2026-10-21 17:21:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2667 [2026-10-21 17:21:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6124 [2026-10-21 17:21:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3864 [2026-10-21 17:21:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1153 [2026-10-21 17:21:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9912 [2026-10-21 17:21:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1356 [2026-10-21 17:21:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7598 [2026-10-21 17:21:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9535 [2026-10-21 17:21:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3078 [2026-10-21 17:21:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3192 [2026-10-21 17:21:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2729 [2026-10-21 17:21:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8357 [2026-10-21 17:22:03][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,612 | MFU: 1.8% [2026-10-21 17:22:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2604 [2026-10-21 17:22:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2449 [2026-10-21 17:22:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8990 [2026-10-21 17:22:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0635 [2026-10-21 17:22:08][INFO][training.pretrain_engine] [Ph2|Step 265] loss=5.1759 | ppl=176.95 | aux=0.02507 | lr=2.99e-04 | gnorm=0.527 | tok/s=29,382 | MFU=7.1% | step=142753ms | tokens=1.080B | mem=4.2/32.7GB | cost=$21.72/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 04:58 (131.6h) | Phase3: SKIPPED | SFT: Oct 27 09:41 (136.3h) | DPO (Final): Oct 27 10:15 (136.9h) [2026-10-21 17:22:08][INFO][training.pretrain_engine] Starting global step 266 (accumulating 32 micro-batches)... [2026-10-21 17:22:08][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 17:22:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5914 [2026-10-21 17:22:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7652 [2026-10-21 17:22:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4654 [2026-10-21 17:22:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8542 [2026-10-21 17:22:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6213 [2026-10-21 17:22:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4689 [2026-10-21 17:22:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7558 [2026-10-21 17:22:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7767 [2026-10-21 17:23:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6002 [2026-10-21 17:23:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0364 [2026-10-21 17:23:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4196 [2026-10-21 17:23:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8454 [2026-10-21 17:23:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3305 [2026-10-21 17:23:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9393 [2026-10-21 17:23:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9673 [2026-10-21 17:23:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1377 [2026-10-21 17:23:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8706 [2026-10-21 17:23:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6092 [2026-10-21 17:23:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9403 [2026-10-21 17:23:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2784 [2026-10-21 17:23:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2962 [2026-10-21 17:23:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3676 [2026-10-21 17:23:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3240 [2026-10-21 17:23:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6228 [2026-10-21 17:24:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9831 [2026-10-21 17:24:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9687 [2026-10-21 17:24:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0450 [2026-10-21 17:24:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3024 [2026-10-21 17:24:26][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,605 | MFU: 1.8% [2026-10-21 17:24:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0960 [2026-10-21 17:24:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9547 [2026-10-21 17:24:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0582 [2026-10-21 17:24:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2822 [2026-10-21 17:24:31][INFO][training.pretrain_engine] [Ph2|Step 266] loss=5.1119 | ppl=165.98 | aux=0.02506 | lr=2.99e-04 | gnorm=0.520 | tok/s=29,380 | MFU=7.1% | step=142759ms | tokens=1.081B | mem=4.2/32.7GB | cost=$21.80/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 05:00 (131.6h) | Phase3: SKIPPED | SFT: Oct 27 09:43 (136.3h) | DPO (Final): Oct 27 10:18 (136.9h) [2026-10-21 17:24:31][INFO][training.pretrain_engine] Starting global step 267 (accumulating 32 micro-batches)... [2026-10-21 17:24:31][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 17:24:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4374 [2026-10-21 17:24:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4305 [2026-10-21 17:24:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7195 [2026-10-21 17:24:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7390 [2026-10-21 17:25:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5131 [2026-10-21 17:25:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3268 [2026-10-21 17:25:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6634 [2026-10-21 17:25:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0506 [2026-10-21 17:25:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6278 [2026-10-21 17:25:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8586 [2026-10-21 17:25:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4465 [2026-10-21 17:25:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0208 [2026-10-21 17:25:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6696 [2026-10-21 17:25:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0223 [2026-10-21 17:25:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9021 [2026-10-21 17:25:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8577 [2026-10-21 17:25:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5009 [2026-10-21 17:26:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7881 [2026-10-21 17:26:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7810 [2026-10-21 17:26:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5252 [2026-10-21 17:26:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9779 [2026-10-21 17:26:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4640 [2026-10-21 17:26:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8804 [2026-10-21 17:26:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3933 [2026-10-21 17:26:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3543 [2026-10-21 17:26:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4605 [2026-10-21 17:26:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3708 [2026-10-21 17:26:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5877 [2026-10-21 17:26:48][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,606 | MFU: 1.8% [2026-10-21 17:26:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1404 [2026-10-21 17:26:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2719 [2026-10-21 17:26:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1853 [2026-10-21 17:26:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8598 [2026-10-21 17:26:53][INFO][training.pretrain_engine] [Ph2|Step 267] loss=5.1922 | ppl=179.87 | aux=0.02506 | lr=2.99e-04 | gnorm=0.465 | tok/s=29,380 | MFU=7.1% | step=142762ms | tokens=1.082B | mem=4.2/32.7GB | cost=$21.88/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 05:02 (131.6h) | Phase3: SKIPPED | SFT: Oct 27 09:45 (136.3h) | DPO (Final): Oct 27 10:19 (136.9h) [2026-10-21 17:26:53][INFO][training.pretrain_engine] Starting global step 268 (accumulating 32 micro-batches)... [2026-10-21 17:26:53][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 17:27:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2778 [2026-10-21 17:27:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0738 [2026-10-21 17:27:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1785 [2026-10-21 17:27:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2895 [2026-10-21 17:27:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5627 [2026-10-21 17:27:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.8356 [2026-10-21 17:27:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9272 [2026-10-21 17:27:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4311 [2026-10-21 17:27:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4662 [2026-10-21 17:27:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8715 [2026-10-21 17:27:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9320 [2026-10-21 17:27:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0248 [2026-10-21 17:28:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5031 [2026-10-21 17:28:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2029 [2026-10-21 17:28:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7682 [2026-10-21 17:28:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0308 [2026-10-21 17:28:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9855 [2026-10-21 17:28:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9352 [2026-10-21 17:28:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0681 [2026-10-21 17:28:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0415 [2026-10-21 17:28:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4412 [2026-10-21 17:28:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6264 [2026-10-21 17:28:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6214 [2026-10-21 17:28:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4017 [2026-10-21 17:28:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3454 [2026-10-21 17:28:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.1259 [2026-10-21 17:28:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4433 [2026-10-21 17:28:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0350 [2026-10-21 17:29:11][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,610 | MFU: 1.8% [2026-10-21 17:29:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0040 [2026-10-21 17:29:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0793 [2026-10-21 17:29:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0994 [2026-10-21 17:29:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9472 [2026-10-21 17:29:16][INFO][training.pretrain_engine] [Ph2|Step 268] loss=5.1792 | ppl=177.54 | aux=0.02506 | lr=2.99e-04 | gnorm=0.453 | tok/s=29,379 | MFU=7.1% | step=142766ms | tokens=1.083B | mem=4.2/32.7GB | cost=$21.96/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 05:04 (131.6h) | Phase3: SKIPPED | SFT: Oct 27 09:47 (136.3h) | DPO (Final): Oct 27 10:21 (136.9h) [2026-10-21 17:29:16][INFO][training.pretrain_engine] Starting global step 269 (accumulating 32 micro-batches)... [2026-10-21 17:29:16][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 17:29:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7300 [2026-10-21 17:29:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7685 [2026-10-21 17:29:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0488 [2026-10-21 17:29:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1547 [2026-10-21 17:29:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9724 [2026-10-21 17:29:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9054 [2026-10-21 17:29:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1813 [2026-10-21 17:29:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0863 [2026-10-21 17:30:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2203 [2026-10-21 17:30:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8894 [2026-10-21 17:30:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9197 [2026-10-21 17:30:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4364 [2026-10-21 17:30:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5501 [2026-10-21 17:30:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7553 [2026-10-21 17:30:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3100 [2026-10-21 17:30:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2544 [2026-10-21 17:30:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2667 [2026-10-21 17:30:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8161 [2026-10-21 17:30:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2447 [2026-10-21 17:30:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1656 [2026-10-21 17:31:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6581 [2026-10-21 17:31:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3684 [2026-10-21 17:31:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7285 [2026-10-21 17:31:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3568 [2026-10-21 17:31:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0940 [2026-10-21 17:31:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4627 [2026-10-21 17:31:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5031 [2026-10-21 17:31:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8779 [2026-10-21 17:31:34][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,588 | MFU: 1.8% [2026-10-21 17:31:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4863 [2026-10-21 17:31:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1908 [2026-10-21 17:31:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7832 [2026-10-21 17:31:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5590 [2026-10-21 17:31:39][INFO][training.pretrain_engine] [Ph2|Step 269] loss=5.1582 | ppl=173.85 | aux=0.02506 | lr=2.99e-04 | gnorm=0.477 | tok/s=29,378 | MFU=7.1% | step=142769ms | tokens=1.084B | mem=4.2/32.7GB | cost=$22.05/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 05:06 (131.6h) | Phase3: SKIPPED | SFT: Oct 27 09:49 (136.3h) | DPO (Final): Oct 27 10:23 (136.9h) [2026-10-21 17:31:39][INFO][training.pretrain_engine] Starting global step 270 (accumulating 32 micro-batches)... [2026-10-21 17:31:39][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 17:31:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5664 [2026-10-21 17:31:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9265 [2026-10-21 17:31:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9732 [2026-10-21 17:31:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3063 [2026-10-21 17:32:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4391 [2026-10-21 17:32:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.8997 [2026-10-21 17:32:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0271 [2026-10-21 17:32:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0979 [2026-10-21 17:32:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1600 [2026-10-21 17:32:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9834 [2026-10-21 17:32:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0730 [2026-10-21 17:32:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0876 [2026-10-21 17:32:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2470 [2026-10-21 17:32:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0957 [2026-10-21 17:32:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3234 [2026-10-21 17:32:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1161 [2026-10-21 17:33:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5810 [2026-10-21 17:33:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1482 [2026-10-21 17:33:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6456 [2026-10-21 17:33:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5389 [2026-10-21 17:33:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3992 [2026-10-21 17:33:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4878 [2026-10-21 17:33:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8470 [2026-10-21 17:33:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6744 [2026-10-21 17:33:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7357 [2026-10-21 17:33:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1316 [2026-10-21 17:33:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0495 [2026-10-21 17:33:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1549 [2026-10-21 17:33:57][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,607 | MFU: 1.8% [2026-10-21 17:34:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3948 [2026-10-21 17:34:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2990 [2026-10-21 17:34:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1672 [2026-10-21 17:34:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7057 [2026-10-21 17:34:02][INFO][training.pretrain_engine] [Ph2|Step 270] loss=5.0368 | ppl=153.98 | aux=0.02507 | lr=2.99e-04 | gnorm=0.602 | tok/s=29,378 | MFU=7.1% | step=142772ms | tokens=1.085B | mem=4.2/32.7GB | cost=$22.13/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 05:08 (131.6h) | Phase3: SKIPPED | SFT: Oct 27 09:51 (136.3h) | DPO (Final): Oct 27 10:25 (136.9h) [2026-10-21 17:34:05][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step270 | Tokens: 1.085B | Spent: $22.13 | Remaining: $977.87 [2026-10-21 17:34:05][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step270 [2026-10-21 17:34:05][INFO][training.pretrain_engine] Starting global step 271 (accumulating 32 micro-batches)... [2026-10-21 17:34:05][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 17:34:05][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 17:34:05][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 17:34:05][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 17:34:05][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step270 to Phase-Technologies/forge-500m-pretrain (path: phase2_step270)... [2026-10-21 17:34:05][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 17:34:15][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 17:34:15][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step270 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 17:34:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7465 [2026-10-21 17:34:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3776 [2026-10-21 17:34:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1043 [2026-10-21 17:34:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1130 [2026-10-21 17:34:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3439 [2026-10-21 17:34:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6880 [2026-10-21 17:34:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6183 [2026-10-21 17:34:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1773 [2026-10-21 17:34:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9941 [2026-10-21 17:34:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0508 [2026-10-21 17:34:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4756 [2026-10-21 17:34:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7156 [2026-10-21 17:35:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8743 [2026-10-21 17:35:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6402 [2026-10-21 17:35:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1318 [2026-10-21 17:35:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2018 [2026-10-21 17:35:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0616 [2026-10-21 17:35:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1621 [2026-10-21 17:35:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3299 [2026-10-21 17:35:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8394 [2026-10-21 17:35:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6357 [2026-10-21 17:35:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3662 [2026-10-21 17:35:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4257 [2026-10-21 17:35:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6145 [2026-10-21 17:36:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5099 [2026-10-21 17:36:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3823 [2026-10-21 17:36:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5735 [2026-10-21 17:36:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8933 [2026-10-21 17:36:23][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,594 | MFU: 1.8% [2026-10-21 17:36:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1390 [2026-10-21 17:36:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6856 [2026-10-21 17:36:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1877 [2026-10-21 17:36:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3347 [2026-10-21 17:36:28][INFO][training.pretrain_engine] [Ph2|Step 271] loss=5.1464 | ppl=171.82 | aux=0.02506 | lr=2.99e-04 | gnorm=0.738 | tok/s=29,378 | MFU=7.1% | step=142771ms | tokens=1.086B | mem=4.2/32.7GB | cost=$22.21/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 05:10 (131.6h) | Phase3: SKIPPED | SFT: Oct 27 09:53 (136.3h) | DPO (Final): Oct 27 10:27 (136.9h) [2026-10-21 17:36:28][INFO][training.pretrain_engine] Starting global step 272 (accumulating 32 micro-batches)... [2026-10-21 17:36:28][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 17:36:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0516 [2026-10-21 17:36:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1314 [2026-10-21 17:36:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8376 [2026-10-21 17:36:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5565 [2026-10-21 17:37:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4823 [2026-10-21 17:37:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2637 [2026-10-21 17:37:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0450 [2026-10-21 17:37:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7093 [2026-10-21 17:37:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1765 [2026-10-21 17:37:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6592 [2026-10-21 17:37:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7578 [2026-10-21 17:37:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2306 [2026-10-21 17:37:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5641 [2026-10-21 17:37:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2482 [2026-10-21 17:37:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7853 [2026-10-21 17:37:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0462 [2026-10-21 17:37:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3814 [2026-10-21 17:37:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8490 [2026-10-21 17:37:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9122 [2026-10-21 17:37:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1622 [2026-10-21 17:38:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9193 [2026-10-21 17:38:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4367 [2026-10-21 17:38:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5792 [2026-10-21 17:38:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2201 [2026-10-21 17:38:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1137 [2026-10-21 17:38:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7344 [2026-10-21 17:38:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1306 [2026-10-21 17:38:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2330 [2026-10-21 17:38:45][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,611 | MFU: 1.8% [2026-10-21 17:38:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3589 [2026-10-21 17:38:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1486 [2026-10-21 17:38:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2060 [2026-10-21 17:38:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4313 [2026-10-21 17:38:51][INFO][training.pretrain_engine] [Ph2|Step 272] loss=5.2271 | ppl=186.25 | aux=0.02505 | lr=2.99e-04 | gnorm=1.047 | tok/s=29,376 | MFU=7.1% | step=142782ms | tokens=1.087B | mem=4.2/32.7GB | cost=$22.29/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 05:12 (131.6h) | Phase3: SKIPPED | SFT: Oct 27 09:56 (136.3h) | DPO (Final): Oct 27 10:30 (136.9h) [2026-10-21 17:38:51][INFO][training.pretrain_engine] Starting global step 273 (accumulating 32 micro-batches)... [2026-10-21 17:38:51][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 17:39:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5196 [2026-10-21 17:39:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7629 [2026-10-21 17:39:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4341 [2026-10-21 17:39:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7411 [2026-10-21 17:39:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1862 [2026-10-21 17:39:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1484 [2026-10-21 17:39:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4945 [2026-10-21 17:39:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8970 [2026-10-21 17:39:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8846 [2026-10-21 17:39:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6060 [2026-10-21 17:39:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9537 [2026-10-21 17:39:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2533 [2026-10-21 17:40:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5900 [2026-10-21 17:40:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2678 [2026-10-21 17:40:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4385 [2026-10-21 17:40:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2855 [2026-10-21 17:40:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6501 [2026-10-21 17:40:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4860 [2026-10-21 17:40:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5131 [2026-10-21 17:40:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8996 [2026-10-21 17:40:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9470 [2026-10-21 17:40:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7966 [2026-10-21 17:40:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2362 [2026-10-21 17:40:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3254 [2026-10-21 17:40:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1728 [2026-10-21 17:40:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7175 [2026-10-21 17:40:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6619 [2026-10-21 17:40:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3097 [2026-10-21 17:41:09][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,610 | MFU: 1.8% [2026-10-21 17:41:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9578 [2026-10-21 17:41:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6360 [2026-10-21 17:41:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8172 [2026-10-21 17:41:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9923 [2026-10-21 17:41:14][INFO][training.pretrain_engine] [Ph2|Step 273] loss=5.2258 | ppl=186.00 | aux=0.02506 | lr=2.99e-04 | gnorm=1.180 | tok/s=29,375 | MFU=7.1% | step=142783ms | tokens=1.088B | mem=4.2/32.7GB | cost=$22.38/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 05:14 (131.5h) | Phase3: SKIPPED | SFT: Oct 27 09:57 (136.3h) | DPO (Final): Oct 27 10:31 (136.8h) [2026-10-21 17:41:14][INFO][training.pretrain_engine] Starting global step 274 (accumulating 32 micro-batches)... [2026-10-21 17:41:14][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 17:41:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3281 [2026-10-21 17:41:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6181 [2026-10-21 17:41:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6155 [2026-10-21 17:41:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3165 [2026-10-21 17:41:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5281 [2026-10-21 17:41:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0857 [2026-10-21 17:41:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2420 [2026-10-21 17:41:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2490 [2026-10-21 17:42:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6232 [2026-10-21 17:42:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4338 [2026-10-21 17:42:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1805 [2026-10-21 17:42:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6653 [2026-10-21 17:42:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5981 [2026-10-21 17:42:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3797 [2026-10-21 17:42:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9425 [2026-10-21 17:42:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6576 [2026-10-21 17:42:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8411 [2026-10-21 17:42:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9048 [2026-10-21 17:42:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0585 [2026-10-21 17:42:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4767 [2026-10-21 17:43:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0399 [2026-10-21 17:43:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=3.9555 [2026-10-21 17:43:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0771 [2026-10-21 17:43:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9892 [2026-10-21 17:43:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2921 [2026-10-21 17:43:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0460 [2026-10-21 17:43:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3080 [2026-10-21 17:43:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1652 [2026-10-21 17:43:32][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,617 | MFU: 1.8% [2026-10-21 17:43:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2110 [2026-10-21 17:43:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7443 [2026-10-21 17:43:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9403 [2026-10-21 17:43:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2747 [2026-10-21 17:43:37][INFO][training.pretrain_engine] [Ph2|Step 274] loss=4.9533 | ppl=141.65 | aux=0.02507 | lr=2.99e-04 | gnorm=0.746 | tok/s=29,375 | MFU=7.1% | step=142787ms | tokens=1.089B | mem=4.2/32.7GB | cost=$22.46/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 05:16 (131.5h) | Phase3: SKIPPED | SFT: Oct 27 09:59 (136.3h) | DPO (Final): Oct 27 10:33 (136.8h) [2026-10-21 17:43:37][INFO][training.pretrain_engine] Starting global step 275 (accumulating 32 micro-batches)... [2026-10-21 17:43:37][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 17:43:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5360 [2026-10-21 17:43:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1290 [2026-10-21 17:43:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7524 [2026-10-21 17:43:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1969 [2026-10-21 17:44:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3939 [2026-10-21 17:44:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1696 [2026-10-21 17:44:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9236 [2026-10-21 17:44:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0210 [2026-10-21 17:44:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2433 [2026-10-21 17:44:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1673 [2026-10-21 17:44:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0462 [2026-10-21 17:44:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1624 [2026-10-21 17:44:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3573 [2026-10-21 17:44:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1248 [2026-10-21 17:44:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7856 [2026-10-21 17:44:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1945 [2026-10-21 17:45:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1869 [2026-10-21 17:45:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3432 [2026-10-21 17:45:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6495 [2026-10-21 17:45:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6865 [2026-10-21 17:45:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0922 [2026-10-21 17:45:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3801 [2026-10-21 17:45:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0513 [2026-10-21 17:45:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3220 [2026-10-21 17:45:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9775 [2026-10-21 17:45:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5169 [2026-10-21 17:45:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3730 [2026-10-21 17:45:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7056 [2026-10-21 17:45:55][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,610 | MFU: 1.8% [2026-10-21 17:46:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8107 [2026-10-21 17:46:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8558 [2026-10-21 17:46:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9506 [2026-10-21 17:46:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9643 [2026-10-21 17:46:00][INFO][training.pretrain_engine] [Ph2|Step 275] loss=5.1265 | ppl=168.43 | aux=0.02506 | lr=2.99e-04 | gnorm=0.961 | tok/s=29,373 | MFU=7.0% | step=142792ms | tokens=1.091B | mem=4.2/32.7GB | cost=$22.54/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 05:18 (131.5h) | Phase3: SKIPPED | SFT: Oct 27 10:02 (136.3h) | DPO (Final): Oct 27 10:36 (136.8h) [2026-10-21 17:46:00][INFO][training.pretrain_engine] Starting global step 276 (accumulating 32 micro-batches)... [2026-10-21 17:46:00][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 17:46:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2140 [2026-10-21 17:46:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5378 [2026-10-21 17:46:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0769 [2026-10-21 17:46:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2353 [2026-10-21 17:46:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9650 [2026-10-21 17:46:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2462 [2026-10-21 17:46:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9285 [2026-10-21 17:46:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6407 [2026-10-21 17:46:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5967 [2026-10-21 17:46:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6504 [2026-10-21 17:46:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1099 [2026-10-21 17:46:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4399 [2026-10-21 17:47:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1122 [2026-10-21 17:47:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0739 [2026-10-21 17:47:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0593 [2026-10-21 17:47:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8712 [2026-10-21 17:47:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1243 [2026-10-21 17:47:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7038 [2026-10-21 17:47:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0540 [2026-10-21 17:47:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5335 [2026-10-21 17:47:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0036 [2026-10-21 17:47:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8700 [2026-10-21 17:47:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3504 [2026-10-21 17:47:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9088 [2026-10-21 17:48:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2267 [2026-10-21 17:48:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9069 [2026-10-21 17:48:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4695 [2026-10-21 17:48:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2568 [2026-10-21 17:48:18][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,610 | MFU: 1.8% [2026-10-21 17:48:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1251 [2026-10-21 17:48:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6762 [2026-10-21 17:48:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0057 [2026-10-21 17:48:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3650 [2026-10-21 17:48:23][INFO][training.pretrain_engine] [Ph2|Step 276] loss=5.1274 | ppl=168.59 | aux=0.02506 | lr=2.99e-04 | gnorm=0.691 | tok/s=29,374 | MFU=7.0% | step=142791ms | tokens=1.092B | mem=4.2/32.7GB | cost=$22.62/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 05:20 (131.5h) | Phase3: SKIPPED | SFT: Oct 27 10:03 (136.3h) | DPO (Final): Oct 27 10:37 (136.8h) [2026-10-21 17:48:23][INFO][training.pretrain_engine] Starting global step 277 (accumulating 32 micro-batches)... [2026-10-21 17:48:23][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 17:48:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2937 [2026-10-21 17:48:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1376 [2026-10-21 17:48:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6071 [2026-10-21 17:48:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5504 [2026-10-21 17:48:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1009 [2026-10-21 17:48:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6327 [2026-10-21 17:48:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4348 [2026-10-21 17:48:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4866 [2026-10-21 17:49:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2690 [2026-10-21 17:49:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2238 [2026-10-21 17:49:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3946 [2026-10-21 17:49:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1746 [2026-10-21 17:49:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0347 [2026-10-21 17:49:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7338 [2026-10-21 17:49:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7973 [2026-10-21 17:49:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8415 [2026-10-21 17:49:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5771 [2026-10-21 17:49:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4415 [2026-10-21 17:49:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4923 [2026-10-21 17:49:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3413 [2026-10-21 17:50:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9677 [2026-10-21 17:50:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0831 [2026-10-21 17:50:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6156 [2026-10-21 17:50:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2493 [2026-10-21 17:50:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2243 [2026-10-21 17:50:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9423 [2026-10-21 17:50:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6812 [2026-10-21 17:50:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8361 [2026-10-21 17:50:41][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,611 | MFU: 1.8% [2026-10-21 17:50:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4292 [2026-10-21 17:50:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3403 [2026-10-21 17:50:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6863 [2026-10-21 17:50:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3425 [2026-10-21 17:50:46][INFO][training.pretrain_engine] [Ph2|Step 277] loss=5.2646 | ppl=193.37 | aux=0.02506 | lr=2.99e-04 | gnorm=0.699 | tok/s=29,374 | MFU=7.0% | step=142792ms | tokens=1.093B | mem=4.2/32.7GB | cost=$22.70/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 05:21 (131.5h) | Phase3: SKIPPED | SFT: Oct 27 10:05 (136.2h) | DPO (Final): Oct 27 10:39 (136.8h) [2026-10-21 17:50:46][INFO][training.pretrain_engine] Starting global step 278 (accumulating 32 micro-batches)... [2026-10-21 17:50:46][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 17:51:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7244 [2026-10-21 17:51:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7478 [2026-10-21 17:51:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1787 [2026-10-21 17:51:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4068 [2026-10-21 17:51:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4732 [2026-10-21 17:51:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2761 [2026-10-21 17:51:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6848 [2026-10-21 17:51:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4296 [2026-10-21 17:51:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8286 [2026-10-21 17:51:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5367 [2026-10-21 17:51:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3342 [2026-10-21 17:51:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2199 [2026-10-21 17:51:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9361 [2026-10-21 17:51:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3426 [2026-10-21 17:51:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5854 [2026-10-21 17:51:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7114 [2026-10-21 17:52:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2702 [2026-10-21 17:52:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5618 [2026-10-21 17:52:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1213 [2026-10-21 17:52:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1125 [2026-10-21 17:52:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2858 [2026-10-21 17:52:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1173 [2026-10-21 17:52:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1999 [2026-10-21 17:52:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0978 [2026-10-21 17:52:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5258 [2026-10-21 17:52:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6131 [2026-10-21 17:52:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7348 [2026-10-21 17:52:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6915 [2026-10-21 17:53:04][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,603 | MFU: 1.8% [2026-10-21 17:53:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8428 [2026-10-21 17:53:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4264 [2026-10-21 17:53:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5859 [2026-10-21 17:53:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0618 [2026-10-21 17:53:09][INFO][training.pretrain_engine] [Ph2|Step 278] loss=5.1501 | ppl=172.45 | aux=0.02507 | lr=2.99e-04 | gnorm=0.621 | tok/s=29,373 | MFU=7.0% | step=142792ms | tokens=1.094B | mem=4.2/32.7GB | cost=$22.78/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 05:23 (131.5h) | Phase3: SKIPPED | SFT: Oct 27 10:07 (136.2h) | DPO (Final): Oct 27 10:41 (136.8h) [2026-10-21 17:53:09][INFO][training.pretrain_engine] Starting global step 279 (accumulating 32 micro-batches)... [2026-10-21 17:53:09][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 17:53:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2889 [2026-10-21 17:53:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7712 [2026-10-21 17:53:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0888 [2026-10-21 17:53:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0927 [2026-10-21 17:53:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3736 [2026-10-21 17:53:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2882 [2026-10-21 17:53:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1906 [2026-10-21 17:53:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9184 [2026-10-21 17:54:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6783 [2026-10-21 17:54:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6727 [2026-10-21 17:54:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8992 [2026-10-21 17:54:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8097 [2026-10-21 17:54:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8049 [2026-10-21 17:54:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1843 [2026-10-21 17:54:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8648 [2026-10-21 17:54:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7888 [2026-10-21 17:54:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4135 [2026-10-21 17:54:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2567 [2026-10-21 17:54:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1317 [2026-10-21 17:54:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5199 [2026-10-21 17:54:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8028 [2026-10-21 17:54:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4357 [2026-10-21 17:54:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2347 [2026-10-21 17:54:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1005 [2026-10-21 17:55:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4343 [2026-10-21 17:55:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3327 [2026-10-21 17:55:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5068 [2026-10-21 17:55:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4778 [2026-10-21 17:55:27][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,605 | MFU: 1.8% [2026-10-21 17:55:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1486 [2026-10-21 17:55:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2443 [2026-10-21 17:55:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5597 [2026-10-21 17:55:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1889 [2026-10-21 17:55:32][INFO][training.pretrain_engine] [Ph2|Step 279] loss=5.2000 | ppl=181.27 | aux=0.02507 | lr=2.99e-04 | gnorm=0.578 | tok/s=29,373 | MFU=7.0% | step=142794ms | tokens=1.095B | mem=4.2/32.7GB | cost=$22.87/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 05:25 (131.5h) | Phase3: SKIPPED | SFT: Oct 27 10:09 (136.2h) | DPO (Final): Oct 27 10:43 (136.8h) [2026-10-21 17:55:35][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step279 | Tokens: 1.095B | Spent: $22.87 | Remaining: $977.13 [2026-10-21 17:55:35][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step279 [2026-10-21 17:55:35][INFO][training.pretrain_engine] Starting global step 280 (accumulating 32 micro-batches)... [2026-10-21 17:55:35][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 17:55:35][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 17:55:35][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 17:55:35][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 17:55:35][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step279 to Phase-Technologies/forge-500m-pretrain (path: phase2_step279)... [2026-10-21 17:55:35][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 17:55:44][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 17:55:44][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step279 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 17:55:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3221 [2026-10-21 17:55:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3058 [2026-10-21 17:55:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1635 [2026-10-21 17:55:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2015 [2026-10-21 17:56:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2941 [2026-10-21 17:56:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2578 [2026-10-21 17:56:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1445 [2026-10-21 17:56:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9777 [2026-10-21 17:56:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2608 [2026-10-21 17:56:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6588 [2026-10-21 17:56:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5570 [2026-10-21 17:56:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7532 [2026-10-21 17:56:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7936 [2026-10-21 17:56:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7464 [2026-10-21 17:56:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3608 [2026-10-21 17:56:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2965 [2026-10-21 17:57:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5895 [2026-10-21 17:57:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8406 [2026-10-21 17:57:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3797 [2026-10-21 17:57:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5620 [2026-10-21 17:57:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8221 [2026-10-21 17:57:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0632 [2026-10-21 17:57:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8957 [2026-10-21 17:57:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9788 [2026-10-21 17:57:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1102 [2026-10-21 17:57:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4098 [2026-10-21 17:57:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2809 [2026-10-21 17:57:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2897 [2026-10-21 17:57:52][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,607 | MFU: 1.8% [2026-10-21 17:57:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2749 [2026-10-21 17:57:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0652 [2026-10-21 17:57:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1225 [2026-10-21 17:57:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6291 [2026-10-21 17:57:57][INFO][training.pretrain_engine] [Ph2|Step 280] loss=5.2052 | ppl=182.21 | aux=0.02506 | lr=2.99e-04 | gnorm=0.566 | tok/s=29,375 | MFU=7.1% | step=142787ms | tokens=1.096B | mem=4.2/32.7GB | cost=$22.95/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 05:26 (131.5h) | Phase3: SKIPPED | SFT: Oct 27 10:10 (136.2h) | DPO (Final): Oct 27 10:44 (136.8h) [2026-10-21 17:57:57][INFO][training.pretrain_engine] Starting global step 281 (accumulating 32 micro-batches)... [2026-10-21 17:57:57][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 17:58:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8140 [2026-10-21 17:58:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5231 [2026-10-21 17:58:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0389 [2026-10-21 17:58:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6865 [2026-10-21 17:58:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2077 [2026-10-21 17:58:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8641 [2026-10-21 17:58:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9930 [2026-10-21 17:58:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3765 [2026-10-21 17:58:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8314 [2026-10-21 17:58:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4688 [2026-10-21 17:58:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9589 [2026-10-21 17:58:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4568 [2026-10-21 17:59:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9043 [2026-10-21 17:59:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0297 [2026-10-21 17:59:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2501 [2026-10-21 17:59:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6079 [2026-10-21 17:59:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3765 [2026-10-21 17:59:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0530 [2026-10-21 17:59:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2330 [2026-10-21 17:59:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8465 [2026-10-21 17:59:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9450 [2026-10-21 17:59:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7752 [2026-10-21 17:59:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8157 [2026-10-21 17:59:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2714 [2026-10-21 18:00:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0004 [2026-10-21 18:00:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1152 [2026-10-21 18:00:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1926 [2026-10-21 18:00:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4942 [2026-10-21 18:00:15][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,613 | MFU: 1.8% [2026-10-21 18:00:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4112 [2026-10-21 18:00:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5598 [2026-10-21 18:00:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2777 [2026-10-21 18:00:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2392 [2026-10-21 18:00:19][INFO][training.pretrain_engine] [Ph2|Step 281] loss=5.0884 | ppl=162.13 | aux=0.02508 | lr=2.99e-04 | gnorm=0.496 | tok/s=29,374 | MFU=7.0% | step=142790ms | tokens=1.097B | mem=4.2/32.7GB | cost=$23.03/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 05:28 (131.5h) | Phase3: SKIPPED | SFT: Oct 27 10:12 (136.2h) | DPO (Final): Oct 27 10:46 (136.8h) [2026-10-21 18:00:19][INFO][training.pretrain_engine] Starting global step 282 (accumulating 32 micro-batches)... [2026-10-21 18:00:19][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 18:00:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0792 [2026-10-21 18:00:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0528 [2026-10-21 18:00:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2083 [2026-10-21 18:00:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0983 [2026-10-21 18:00:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5631 [2026-10-21 18:00:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0164 [2026-10-21 18:00:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8749 [2026-10-21 18:00:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9428 [2026-10-21 18:01:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8566 [2026-10-21 18:01:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9631 [2026-10-21 18:01:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9520 [2026-10-21 18:01:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0207 [2026-10-21 18:01:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5595 [2026-10-21 18:01:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=3.5747 [2026-10-21 18:01:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4773 [2026-10-21 18:01:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5079 [2026-10-21 18:01:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2208 [2026-10-21 18:01:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8959 [2026-10-21 18:01:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1407 [2026-10-21 18:01:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1259 [2026-10-21 18:02:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6858 [2026-10-21 18:02:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0865 [2026-10-21 18:02:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5221 [2026-10-21 18:02:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2865 [2026-10-21 18:02:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7114 [2026-10-21 18:02:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1899 [2026-10-21 18:02:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3077 [2026-10-21 18:02:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9400 [2026-10-21 18:02:37][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,618 | MFU: 1.8% [2026-10-21 18:02:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0410 [2026-10-21 18:02:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9287 [2026-10-21 18:02:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9697 [2026-10-21 18:02:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1773 [2026-10-21 18:02:42][INFO][training.pretrain_engine] [Ph2|Step 282] loss=4.9884 | ppl=146.70 | aux=0.02507 | lr=2.99e-04 | gnorm=0.594 | tok/s=29,374 | MFU=7.1% | step=142789ms | tokens=1.098B | mem=4.2/32.7GB | cost=$23.11/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 05:30 (131.5h) | Phase3: SKIPPED | SFT: Oct 27 10:14 (136.2h) | DPO (Final): Oct 27 10:48 (136.8h) [2026-10-21 18:02:42][INFO][training.pretrain_engine] Starting global step 283 (accumulating 32 micro-batches)... [2026-10-21 18:02:42][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 18:03:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7397 [2026-10-21 18:03:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4900 [2026-10-21 18:03:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0914 [2026-10-21 18:03:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2656 [2026-10-21 18:03:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0402 [2026-10-21 18:03:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7779 [2026-10-21 18:03:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0647 [2026-10-21 18:03:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0573 [2026-10-21 18:03:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2226 [2026-10-21 18:03:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1274 [2026-10-21 18:03:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4705 [2026-10-21 18:03:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5037 [2026-10-21 18:03:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2612 [2026-10-21 18:03:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0667 [2026-10-21 18:03:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0112 [2026-10-21 18:03:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1089 [2026-10-21 18:04:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2631 [2026-10-21 18:04:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4269 [2026-10-21 18:04:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3264 [2026-10-21 18:04:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3593 [2026-10-21 18:04:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1104 [2026-10-21 18:04:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1399 [2026-10-21 18:04:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1623 [2026-10-21 18:04:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4764 [2026-10-21 18:04:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6613 [2026-10-21 18:04:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6982 [2026-10-21 18:04:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3291 [2026-10-21 18:04:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4256 [2026-10-21 18:05:00][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,629 | MFU: 1.8% [2026-10-21 18:05:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3956 [2026-10-21 18:05:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3008 [2026-10-21 18:05:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3558 [2026-10-21 18:05:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4414 [2026-10-21 18:05:05][INFO][training.pretrain_engine] [Ph2|Step 283] loss=5.0597 | ppl=157.55 | aux=0.02507 | lr=2.99e-04 | gnorm=0.531 | tok/s=29,375 | MFU=7.1% | step=142785ms | tokens=1.099B | mem=4.2/32.7GB | cost=$23.19/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 05:32 (131.5h) | Phase3: SKIPPED | SFT: Oct 27 10:15 (136.2h) | DPO (Final): Oct 27 10:49 (136.7h) [2026-10-21 18:05:05][INFO][training.pretrain_engine] Starting global step 284 (accumulating 32 micro-batches)... [2026-10-21 18:05:05][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 18:05:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6350 [2026-10-21 18:05:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4367 [2026-10-21 18:05:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9639 [2026-10-21 18:05:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8010 [2026-10-21 18:05:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2680 [2026-10-21 18:05:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2047 [2026-10-21 18:05:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0087 [2026-10-21 18:05:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6407 [2026-10-21 18:05:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2316 [2026-10-21 18:05:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8379 [2026-10-21 18:05:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7389 [2026-10-21 18:05:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7319 [2026-10-21 18:06:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6618 [2026-10-21 18:06:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2184 [2026-10-21 18:06:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4020 [2026-10-21 18:06:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0652 [2026-10-21 18:06:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3044 [2026-10-21 18:06:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3193 [2026-10-21 18:06:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2402 [2026-10-21 18:06:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1277 [2026-10-21 18:06:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8392 [2026-10-21 18:06:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4480 [2026-10-21 18:06:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2485 [2026-10-21 18:06:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6844 [2026-10-21 18:07:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7386 [2026-10-21 18:07:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1010 [2026-10-21 18:07:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3132 [2026-10-21 18:07:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9389 [2026-10-21 18:07:23][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,616 | MFU: 1.8% [2026-10-21 18:07:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5859 [2026-10-21 18:07:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5974 [2026-10-21 18:07:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2831 [2026-10-21 18:07:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3485 [2026-10-21 18:07:28][INFO][training.pretrain_engine] [Ph2|Step 284] loss=5.0887 | ppl=162.18 | aux=0.02507 | lr=2.99e-04 | gnorm=0.598 | tok/s=29,375 | MFU=7.1% | step=142784ms | tokens=1.100B | mem=4.2/32.7GB | cost=$23.28/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 05:33 (131.4h) | Phase3: SKIPPED | SFT: Oct 27 10:17 (136.2h) | DPO (Final): Oct 27 10:51 (136.7h) [2026-10-21 18:07:28][INFO][training.pretrain_engine] Starting global step 285 (accumulating 32 micro-batches)... [2026-10-21 18:07:28][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 18:07:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2302 [2026-10-21 18:07:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5205 [2026-10-21 18:07:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2001 [2026-10-21 18:07:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2085 [2026-10-21 18:08:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7013 [2026-10-21 18:08:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9751 [2026-10-21 18:08:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3678 [2026-10-21 18:08:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0486 [2026-10-21 18:08:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2599 [2026-10-21 18:08:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1033 [2026-10-21 18:08:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3488 [2026-10-21 18:08:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7332 [2026-10-21 18:08:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5070 [2026-10-21 18:08:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4632 [2026-10-21 18:08:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2373 [2026-10-21 18:08:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2832 [2026-10-21 18:08:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2597 [2026-10-21 18:08:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1727 [2026-10-21 18:08:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5850 [2026-10-21 18:08:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1975 [2026-10-21 18:09:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6681 [2026-10-21 18:09:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3232 [2026-10-21 18:09:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9919 [2026-10-21 18:09:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8844 [2026-10-21 18:09:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4112 [2026-10-21 18:09:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4315 [2026-10-21 18:09:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3760 [2026-10-21 18:09:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2681 [2026-10-21 18:09:46][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,612 | MFU: 1.8% [2026-10-21 18:09:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1490 [2026-10-21 18:09:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4306 [2026-10-21 18:09:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2140 [2026-10-21 18:09:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2005 [2026-10-21 18:09:51][INFO][training.pretrain_engine] [Ph2|Step 285] loss=5.1741 | ppl=176.65 | aux=0.02507 | lr=2.99e-04 | gnorm=0.660 | tok/s=29,375 | MFU=7.1% | step=142786ms | tokens=1.101B | mem=4.2/32.7GB | cost=$23.36/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 05:35 (131.4h) | Phase3: SKIPPED | SFT: Oct 27 10:19 (136.2h) | DPO (Final): Oct 27 10:53 (136.7h) [2026-10-21 18:09:51][INFO][training.pretrain_engine] Starting global step 286 (accumulating 32 micro-batches)... [2026-10-21 18:09:51][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 18:10:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6050 [2026-10-21 18:10:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5665 [2026-10-21 18:10:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8501 [2026-10-21 18:10:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2323 [2026-10-21 18:10:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0574 [2026-10-21 18:10:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5280 [2026-10-21 18:10:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8692 [2026-10-21 18:10:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3899 [2026-10-21 18:10:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1395 [2026-10-21 18:10:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6030 [2026-10-21 18:10:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1745 [2026-10-21 18:10:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4678 [2026-10-21 18:11:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9284 [2026-10-21 18:11:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6027 [2026-10-21 18:11:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1920 [2026-10-21 18:11:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9772 [2026-10-21 18:11:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9434 [2026-10-21 18:11:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3030 [2026-10-21 18:11:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3003 [2026-10-21 18:11:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1395 [2026-10-21 18:11:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3720 [2026-10-21 18:11:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3327 [2026-10-21 18:11:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2771 [2026-10-21 18:11:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4836 [2026-10-21 18:11:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4342 [2026-10-21 18:11:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3580 [2026-10-21 18:11:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4718 [2026-10-21 18:11:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3734 [2026-10-21 18:12:08][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,613 | MFU: 1.8% [2026-10-21 18:12:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6287 [2026-10-21 18:12:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7752 [2026-10-21 18:12:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6683 [2026-10-21 18:12:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6916 [2026-10-21 18:12:13][INFO][training.pretrain_engine] [Ph2|Step 286] loss=5.2144 | ppl=183.90 | aux=0.02507 | lr=2.99e-04 | gnorm=0.707 | tok/s=29,376 | MFU=7.1% | step=142781ms | tokens=1.102B | mem=4.2/32.7GB | cost=$23.44/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 05:37 (131.4h) | Phase3: SKIPPED | SFT: Oct 27 10:21 (136.1h) | DPO (Final): Oct 27 10:55 (136.7h) [2026-10-21 18:12:13][INFO][training.pretrain_engine] Starting global step 287 (accumulating 32 micro-batches)... [2026-10-21 18:12:13][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 18:12:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2460 [2026-10-21 18:12:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9583 [2026-10-21 18:12:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2166 [2026-10-21 18:12:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4911 [2026-10-21 18:12:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0458 [2026-10-21 18:12:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0005 [2026-10-21 18:12:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4324 [2026-10-21 18:12:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4387 [2026-10-21 18:13:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6949 [2026-10-21 18:13:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9678 [2026-10-21 18:13:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6029 [2026-10-21 18:13:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9065 [2026-10-21 18:13:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5014 [2026-10-21 18:13:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8340 [2026-10-21 18:13:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9341 [2026-10-21 18:13:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0195 [2026-10-21 18:13:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8305 [2026-10-21 18:13:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8298 [2026-10-21 18:13:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5314 [2026-10-21 18:13:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0287 [2026-10-21 18:13:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5589 [2026-10-21 18:14:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7038 [2026-10-21 18:14:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7768 [2026-10-21 18:14:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6515 [2026-10-21 18:14:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1423 [2026-10-21 18:14:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1460 [2026-10-21 18:14:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1960 [2026-10-21 18:14:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6486 [2026-10-21 18:14:31][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,607 | MFU: 1.8% [2026-10-21 18:14:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0497 [2026-10-21 18:14:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3783 [2026-10-21 18:14:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0397 [2026-10-21 18:14:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2757 [2026-10-21 18:14:36][INFO][training.pretrain_engine] [Ph2|Step 287] loss=5.1210 | ppl=167.50 | aux=0.02506 | lr=2.99e-04 | gnorm=0.867 | tok/s=29,377 | MFU=7.1% | step=142776ms | tokens=1.103B | mem=4.2/32.7GB | cost=$23.52/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 05:38 (131.4h) | Phase3: SKIPPED | SFT: Oct 27 10:22 (136.1h) | DPO (Final): Oct 27 10:56 (136.7h) [2026-10-21 18:14:36][INFO][training.pretrain_engine] Starting global step 288 (accumulating 32 micro-batches)... [2026-10-21 18:14:36][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 18:14:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9216 [2026-10-21 18:14:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3756 [2026-10-21 18:14:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0122 [2026-10-21 18:14:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2279 [2026-10-21 18:15:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9965 [2026-10-21 18:15:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3398 [2026-10-21 18:15:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3013 [2026-10-21 18:15:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8844 [2026-10-21 18:15:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2790 [2026-10-21 18:15:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7411 [2026-10-21 18:15:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1805 [2026-10-21 18:15:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0363 [2026-10-21 18:15:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1678 [2026-10-21 18:15:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1625 [2026-10-21 18:15:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2054 [2026-10-21 18:15:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5095 [2026-10-21 18:16:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1374 [2026-10-21 18:16:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0050 [2026-10-21 18:16:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7415 [2026-10-21 18:16:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8657 [2026-10-21 18:16:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6547 [2026-10-21 18:16:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2684 [2026-10-21 18:16:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0982 [2026-10-21 18:16:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6914 [2026-10-21 18:16:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6606 [2026-10-21 18:16:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3218 [2026-10-21 18:16:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4423 [2026-10-21 18:16:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5576 [2026-10-21 18:16:53][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,616 | MFU: 1.8% [2026-10-21 18:16:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9125 [2026-10-21 18:16:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7276 [2026-10-21 18:16:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7445 [2026-10-21 18:16:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5936 [2026-10-21 18:16:58][INFO][training.pretrain_engine] [Ph2|Step 288] loss=5.0829 | ppl=161.24 | aux=0.02507 | lr=2.99e-04 | gnorm=0.906 | tok/s=29,378 | MFU=7.1% | step=142771ms | tokens=1.104B | mem=4.2/32.7GB | cost=$23.60/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 05:40 (131.4h) | Phase3: SKIPPED | SFT: Oct 27 10:24 (136.1h) | DPO (Final): Oct 27 10:58 (136.7h) [2026-10-21 18:17:01][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step288 | Tokens: 1.104B | Spent: $23.60 | Remaining: $976.40 [2026-10-21 18:17:01][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step288 [2026-10-21 18:17:01][INFO][training.pretrain_engine] Starting global step 289 (accumulating 32 micro-batches)... [2026-10-21 18:17:01][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 18:17:02][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 18:17:02][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 18:17:02][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 18:17:02][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step288 to Phase-Technologies/forge-500m-pretrain (path: phase2_step288)... [2026-10-21 18:17:02][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 18:17:12][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 18:17:12][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step288 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 18:17:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9980 [2026-10-21 18:17:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0059 [2026-10-21 18:17:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6083 [2026-10-21 18:17:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1855 [2026-10-21 18:17:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0394 [2026-10-21 18:17:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4945 [2026-10-21 18:17:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7591 [2026-10-21 18:17:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9662 [2026-10-21 18:17:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4273 [2026-10-21 18:17:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5958 [2026-10-21 18:17:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5583 [2026-10-21 18:17:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8019 [2026-10-21 18:18:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6606 [2026-10-21 18:18:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7136 [2026-10-21 18:18:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1128 [2026-10-21 18:18:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0885 [2026-10-21 18:18:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3545 [2026-10-21 18:18:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6427 [2026-10-21 18:18:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4400 [2026-10-21 18:18:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2552 [2026-10-21 18:18:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5016 [2026-10-21 18:18:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7959 [2026-10-21 18:18:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5694 [2026-10-21 18:18:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4976 [2026-10-21 18:19:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4066 [2026-10-21 18:19:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1033 [2026-10-21 18:19:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0859 [2026-10-21 18:19:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5072 [2026-10-21 18:19:19][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,612 | MFU: 1.8% [2026-10-21 18:19:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8768 [2026-10-21 18:19:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0731 [2026-10-21 18:19:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3312 [2026-10-21 18:19:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1063 [2026-10-21 18:19:24][INFO][training.pretrain_engine] [Ph2|Step 289] loss=5.2306 | ppl=186.91 | aux=0.02506 | lr=2.99e-04 | gnorm=0.859 | tok/s=29,380 | MFU=7.1% | step=142761ms | tokens=1.105B | mem=4.2/32.7GB | cost=$23.69/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 05:41 (131.4h) | Phase3: SKIPPED | SFT: Oct 27 10:25 (136.1h) | DPO (Final): Oct 27 10:59 (136.7h) [2026-10-21 18:19:24][INFO][training.pretrain_engine] Starting global step 290 (accumulating 32 micro-batches)... [2026-10-21 18:19:24][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 18:19:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2848 [2026-10-21 18:19:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1168 [2026-10-21 18:19:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4556 [2026-10-21 18:19:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1207 [2026-10-21 18:19:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5968 [2026-10-21 18:19:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7836 [2026-10-21 18:19:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9621 [2026-10-21 18:19:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8181 [2026-10-21 18:20:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7478 [2026-10-21 18:20:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5347 [2026-10-21 18:20:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4730 [2026-10-21 18:20:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4942 [2026-10-21 18:20:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4580 [2026-10-21 18:20:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2596 [2026-10-21 18:20:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3941 [2026-10-21 18:20:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4569 [2026-10-21 18:20:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1474 [2026-10-21 18:20:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2766 [2026-10-21 18:20:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1852 [2026-10-21 18:20:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1994 [2026-10-21 18:21:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3296 [2026-10-21 18:21:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1684 [2026-10-21 18:21:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6725 [2026-10-21 18:21:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2839 [2026-10-21 18:21:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0586 [2026-10-21 18:21:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1000 [2026-10-21 18:21:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7579 [2026-10-21 18:21:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1338 [2026-10-21 18:21:42][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,597 | MFU: 1.8% [2026-10-21 18:21:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0597 [2026-10-21 18:21:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3702 [2026-10-21 18:21:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7956 [2026-10-21 18:21:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3864 [2026-10-21 18:21:46][INFO][training.pretrain_engine] [Ph2|Step 290] loss=5.1785 | ppl=177.42 | aux=0.02506 | lr=2.99e-04 | gnorm=0.727 | tok/s=29,379 | MFU=7.1% | step=142765ms | tokens=1.106B | mem=4.2/32.7GB | cost=$23.77/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 05:43 (131.4h) | Phase3: SKIPPED | SFT: Oct 27 10:27 (136.1h) | DPO (Final): Oct 27 11:01 (136.7h) [2026-10-21 18:21:46][INFO][training.pretrain_engine] Starting global step 291 (accumulating 32 micro-batches)... [2026-10-21 18:21:46][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 18:22:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2269 [2026-10-21 18:22:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5181 [2026-10-21 18:22:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2069 [2026-10-21 18:22:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3320 [2026-10-21 18:22:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4087 [2026-10-21 18:22:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7943 [2026-10-21 18:22:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3945 [2026-10-21 18:22:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5742 [2026-10-21 18:22:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3359 [2026-10-21 18:22:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3521 [2026-10-21 18:22:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1885 [2026-10-21 18:22:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0042 [2026-10-21 18:22:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7440 [2026-10-21 18:22:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1133 [2026-10-21 18:22:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0026 [2026-10-21 18:22:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0818 [2026-10-21 18:23:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0540 [2026-10-21 18:23:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9603 [2026-10-21 18:23:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1673 [2026-10-21 18:23:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8516 [2026-10-21 18:23:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.0536 [2026-10-21 18:23:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5182 [2026-10-21 18:23:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2662 [2026-10-21 18:23:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3943 [2026-10-21 18:23:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3734 [2026-10-21 18:23:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8500 [2026-10-21 18:23:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4321 [2026-10-21 18:23:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3666 [2026-10-21 18:24:05][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,594 | MFU: 1.8% [2026-10-21 18:24:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4036 [2026-10-21 18:24:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0370 [2026-10-21 18:24:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9773 [2026-10-21 18:24:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9430 [2026-10-21 18:24:09][INFO][training.pretrain_engine] [Ph2|Step 291] loss=5.1744 | ppl=176.68 | aux=0.02505 | lr=2.99e-04 | gnorm=0.816 | tok/s=29,380 | MFU=7.1% | step=142758ms | tokens=1.107B | mem=4.2/32.7GB | cost=$23.85/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 05:45 (131.3h) | Phase3: SKIPPED | SFT: Oct 27 10:28 (136.1h) | DPO (Final): Oct 27 11:02 (136.6h) [2026-10-21 18:24:09][INFO][training.pretrain_engine] Starting global step 292 (accumulating 32 micro-batches)... [2026-10-21 18:24:09][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 18:24:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6675 [2026-10-21 18:24:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1659 [2026-10-21 18:24:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1692 [2026-10-21 18:24:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6931 [2026-10-21 18:24:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0511 [2026-10-21 18:24:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0353 [2026-10-21 18:24:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4586 [2026-10-21 18:24:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4266 [2026-10-21 18:25:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8908 [2026-10-21 18:25:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0408 [2026-10-21 18:25:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=3.8397 [2026-10-21 18:25:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3995 [2026-10-21 18:25:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4395 [2026-10-21 18:25:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3932 [2026-10-21 18:25:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3218 [2026-10-21 18:25:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4001 [2026-10-21 18:25:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0273 [2026-10-21 18:25:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9534 [2026-10-21 18:25:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9362 [2026-10-21 18:25:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0191 [2026-10-21 18:25:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.1188 [2026-10-21 18:25:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2248 [2026-10-21 18:25:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5917 [2026-10-21 18:25:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8040 [2026-10-21 18:26:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3907 [2026-10-21 18:26:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3649 [2026-10-21 18:26:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4319 [2026-10-21 18:26:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3553 [2026-10-21 18:26:27][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,621 | MFU: 1.8% [2026-10-21 18:26:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5684 [2026-10-21 18:26:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2094 [2026-10-21 18:26:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2366 [2026-10-21 18:26:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0060 [2026-10-21 18:26:31][INFO][training.pretrain_engine] [Ph2|Step 292] loss=5.1978 | ppl=180.88 | aux=0.02506 | lr=2.99e-04 | gnorm=0.625 | tok/s=29,382 | MFU=7.1% | step=142752ms | tokens=1.108B | mem=4.2/32.7GB | cost=$23.93/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 05:46 (131.3h) | Phase3: SKIPPED | SFT: Oct 27 10:30 (136.1h) | DPO (Final): Oct 27 11:04 (136.6h) [2026-10-21 18:26:31][INFO][training.pretrain_engine] Starting global step 293 (accumulating 32 micro-batches)... [2026-10-21 18:26:31][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 18:26:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7946 [2026-10-21 18:26:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0823 [2026-10-21 18:26:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9436 [2026-10-21 18:26:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9587 [2026-10-21 18:27:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6239 [2026-10-21 18:27:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9296 [2026-10-21 18:27:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0268 [2026-10-21 18:27:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3221 [2026-10-21 18:27:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4455 [2026-10-21 18:27:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4023 [2026-10-21 18:27:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4024 [2026-10-21 18:27:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1776 [2026-10-21 18:27:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9980 [2026-10-21 18:27:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0691 [2026-10-21 18:27:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8587 [2026-10-21 18:27:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9415 [2026-10-21 18:28:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7247 [2026-10-21 18:28:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7998 [2026-10-21 18:28:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0592 [2026-10-21 18:28:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8106 [2026-10-21 18:28:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3810 [2026-10-21 18:28:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2824 [2026-10-21 18:28:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8767 [2026-10-21 18:28:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6751 [2026-10-21 18:28:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2785 [2026-10-21 18:28:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2886 [2026-10-21 18:28:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1990 [2026-10-21 18:28:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6130 [2026-10-21 18:28:49][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,625 | MFU: 1.8% [2026-10-21 18:28:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2860 [2026-10-21 18:28:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5858 [2026-10-21 18:28:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3429 [2026-10-21 18:28:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=3.8288 [2026-10-21 18:28:54][INFO][training.pretrain_engine] [Ph2|Step 293] loss=5.1646 | ppl=174.97 | aux=0.02506 | lr=2.99e-04 | gnorm=0.578 | tok/s=29,382 | MFU=7.1% | step=142752ms | tokens=1.109B | mem=4.2/32.7GB | cost=$24.01/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 05:48 (131.3h) | Phase3: SKIPPED | SFT: Oct 27 10:31 (136.0h) | DPO (Final): Oct 27 11:05 (136.6h) [2026-10-21 18:28:54][INFO][training.pretrain_engine] Starting global step 294 (accumulating 32 micro-batches)... [2026-10-21 18:28:54][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 18:29:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1186 [2026-10-21 18:29:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2345 [2026-10-21 18:29:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0193 [2026-10-21 18:29:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2929 [2026-10-21 18:29:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1874 [2026-10-21 18:29:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3936 [2026-10-21 18:29:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3140 [2026-10-21 18:29:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7379 [2026-10-21 18:29:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7269 [2026-10-21 18:29:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8845 [2026-10-21 18:29:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7606 [2026-10-21 18:29:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0819 [2026-10-21 18:30:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2246 [2026-10-21 18:30:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0397 [2026-10-21 18:30:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3349 [2026-10-21 18:30:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2835 [2026-10-21 18:30:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3183 [2026-10-21 18:30:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0060 [2026-10-21 18:30:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0488 [2026-10-21 18:30:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0901 [2026-10-21 18:30:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9105 [2026-10-21 18:30:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9524 [2026-10-21 18:30:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2244 [2026-10-21 18:30:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.0275 [2026-10-21 18:30:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8817 [2026-10-21 18:30:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0177 [2026-10-21 18:30:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2441 [2026-10-21 18:30:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0491 [2026-10-21 18:31:11][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,633 | MFU: 1.8% [2026-10-21 18:31:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4128 [2026-10-21 18:31:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4022 [2026-10-21 18:31:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.2431 [2026-10-21 18:31:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9294 [2026-10-21 18:31:16][INFO][training.pretrain_engine] [Ph2|Step 294] loss=5.1891 | ppl=179.31 | aux=0.02507 | lr=2.99e-04 | gnorm=0.582 | tok/s=29,384 | MFU=7.1% | step=142741ms | tokens=1.110B | mem=4.2/32.7GB | cost=$24.09/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 05:49 (131.3h) | Phase3: SKIPPED | SFT: Oct 27 10:33 (136.0h) | DPO (Final): Oct 27 11:07 (136.6h) [2026-10-21 18:31:16][INFO][training.pretrain_engine] Starting global step 295 (accumulating 32 micro-batches)... [2026-10-21 18:31:16][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 18:31:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9012 [2026-10-21 18:31:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2641 [2026-10-21 18:31:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0572 [2026-10-21 18:31:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2544 [2026-10-21 18:31:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1767 [2026-10-21 18:31:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1246 [2026-10-21 18:31:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8759 [2026-10-21 18:31:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0398 [2026-10-21 18:32:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8790 [2026-10-21 18:32:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2334 [2026-10-21 18:32:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5129 [2026-10-21 18:32:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7426 [2026-10-21 18:32:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7609 [2026-10-21 18:32:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9678 [2026-10-21 18:32:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3874 [2026-10-21 18:32:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7998 [2026-10-21 18:32:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3399 [2026-10-21 18:32:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1441 [2026-10-21 18:32:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1379 [2026-10-21 18:32:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7817 [2026-10-21 18:33:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4329 [2026-10-21 18:33:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2965 [2026-10-21 18:33:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2428 [2026-10-21 18:33:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1012 [2026-10-21 18:33:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5193 [2026-10-21 18:33:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4463 [2026-10-21 18:33:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6773 [2026-10-21 18:33:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4167 [2026-10-21 18:33:34][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,624 | MFU: 1.8% [2026-10-21 18:33:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1568 [2026-10-21 18:33:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.1605 [2026-10-21 18:33:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5279 [2026-10-21 18:33:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8396 [2026-10-21 18:33:39][INFO][training.pretrain_engine] [Ph2|Step 295] loss=5.1601 | ppl=174.17 | aux=0.02507 | lr=2.99e-04 | gnorm=0.535 | tok/s=29,385 | MFU=7.1% | step=142738ms | tokens=1.111B | mem=4.2/32.7GB | cost=$24.18/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 05:51 (131.3h) | Phase3: SKIPPED | SFT: Oct 27 10:34 (136.0h) | DPO (Final): Oct 27 11:08 (136.6h) [2026-10-21 18:33:39][INFO][training.pretrain_engine] Starting global step 296 (accumulating 32 micro-batches)... [2026-10-21 18:33:39][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 18:33:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8950 [2026-10-21 18:33:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8917 [2026-10-21 18:33:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0207 [2026-10-21 18:33:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9308 [2026-10-21 18:34:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6866 [2026-10-21 18:34:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6453 [2026-10-21 18:34:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1104 [2026-10-21 18:34:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5032 [2026-10-21 18:34:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6077 [2026-10-21 18:34:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4694 [2026-10-21 18:34:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4023 [2026-10-21 18:34:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6373 [2026-10-21 18:34:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8705 [2026-10-21 18:34:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1618 [2026-10-21 18:34:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1150 [2026-10-21 18:34:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0815 [2026-10-21 18:35:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9007 [2026-10-21 18:35:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8487 [2026-10-21 18:35:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4513 [2026-10-21 18:35:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2228 [2026-10-21 18:35:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5829 [2026-10-21 18:35:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1352 [2026-10-21 18:35:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6258 [2026-10-21 18:35:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4036 [2026-10-21 18:35:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0163 [2026-10-21 18:35:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0254 [2026-10-21 18:35:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2032 [2026-10-21 18:35:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2061 [2026-10-21 18:35:56][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,642 | MFU: 1.8% [2026-10-21 18:36:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1958 [2026-10-21 18:36:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.1582 [2026-10-21 18:36:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3979 [2026-10-21 18:36:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5098 [2026-10-21 18:36:01][INFO][training.pretrain_engine] [Ph2|Step 296] loss=5.0284 | ppl=152.68 | aux=0.02506 | lr=2.99e-04 | gnorm=0.531 | tok/s=29,385 | MFU=7.1% | step=142737ms | tokens=1.113B | mem=4.2/32.7GB | cost=$24.26/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 05:52 (131.3h) | Phase3: SKIPPED | SFT: Oct 27 10:36 (136.0h) | DPO (Final): Oct 27 11:10 (136.6h) [2026-10-21 18:36:01][INFO][training.pretrain_engine] Starting global step 297 (accumulating 32 micro-batches)... [2026-10-21 18:36:01][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 18:36:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9120 [2026-10-21 18:36:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0280 [2026-10-21 18:36:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0275 [2026-10-21 18:36:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2456 [2026-10-21 18:36:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2454 [2026-10-21 18:36:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8743 [2026-10-21 18:36:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2000 [2026-10-21 18:36:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0604 [2026-10-21 18:36:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8016 [2026-10-21 18:36:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1526 [2026-10-21 18:36:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9614 [2026-10-21 18:36:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8831 [2026-10-21 18:37:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0890 [2026-10-21 18:37:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0789 [2026-10-21 18:37:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0920 [2026-10-21 18:37:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1081 [2026-10-21 18:37:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6251 [2026-10-21 18:37:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2286 [2026-10-21 18:37:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9810 [2026-10-21 18:37:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9159 [2026-10-21 18:37:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.1221 [2026-10-21 18:37:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4892 [2026-10-21 18:37:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=3.5561 [2026-10-21 18:37:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2961 [2026-10-21 18:38:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6197 [2026-10-21 18:38:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5726 [2026-10-21 18:38:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4444 [2026-10-21 18:38:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4955 [2026-10-21 18:38:19][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,617 | MFU: 1.8% [2026-10-21 18:38:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0306 [2026-10-21 18:38:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9975 [2026-10-21 18:38:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6402 [2026-10-21 18:38:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5782 [2026-10-21 18:38:24][INFO][training.pretrain_engine] [Ph2|Step 297] loss=5.0491 | ppl=155.88 | aux=0.02507 | lr=2.99e-04 | gnorm=0.516 | tok/s=29,385 | MFU=7.1% | step=142734ms | tokens=1.114B | mem=4.2/32.7GB | cost=$24.34/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 05:54 (131.3h) | Phase3: SKIPPED | SFT: Oct 27 10:38 (136.0h) | DPO (Final): Oct 27 11:12 (136.6h) [2026-10-21 18:38:27][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step297 | Tokens: 1.114B | Spent: $24.34 | Remaining: $975.66 [2026-10-21 18:38:27][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step297 [2026-10-21 18:38:27][INFO][training.pretrain_engine] Starting global step 298 (accumulating 32 micro-batches)... [2026-10-21 18:38:27][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 18:38:27][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 18:38:27][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 18:38:27][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 18:38:27][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step297 to Phase-Technologies/forge-500m-pretrain (path: phase2_step297)... [2026-10-21 18:38:27][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 18:38:39][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 18:38:39][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step297 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 18:38:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4378 [2026-10-21 18:38:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2779 [2026-10-21 18:38:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2586 [2026-10-21 18:38:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2495 [2026-10-21 18:38:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5454 [2026-10-21 18:38:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1709 [2026-10-21 18:38:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5689 [2026-10-21 18:39:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2933 [2026-10-21 18:39:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4912 [2026-10-21 18:39:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2978 [2026-10-21 18:39:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0719 [2026-10-21 18:39:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6763 [2026-10-21 18:39:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0685 [2026-10-21 18:39:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3001 [2026-10-21 18:39:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9398 [2026-10-21 18:39:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4822 [2026-10-21 18:39:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3963 [2026-10-21 18:39:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0834 [2026-10-21 18:39:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9582 [2026-10-21 18:39:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0047 [2026-10-21 18:40:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3033 [2026-10-21 18:40:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1268 [2026-10-21 18:40:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5300 [2026-10-21 18:40:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3735 [2026-10-21 18:40:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2687 [2026-10-21 18:40:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0887 [2026-10-21 18:40:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3492 [2026-10-21 18:40:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1510 [2026-10-21 18:40:44][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,624 | MFU: 1.8% [2026-10-21 18:40:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1566 [2026-10-21 18:40:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9403 [2026-10-21 18:40:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2147 [2026-10-21 18:40:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4733 [2026-10-21 18:40:49][INFO][training.pretrain_engine] [Ph2|Step 298] loss=5.1179 | ppl=166.99 | aux=0.02506 | lr=2.99e-04 | gnorm=0.504 | tok/s=29,387 | MFU=7.1% | step=142727ms | tokens=1.115B | mem=4.2/32.7GB | cost=$24.42/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 05:55 (131.3h) | Phase3: SKIPPED | SFT: Oct 27 10:39 (136.0h) | DPO (Final): Oct 27 11:13 (136.5h) [2026-10-21 18:40:49][INFO][training.pretrain_engine] Starting global step 299 (accumulating 32 micro-batches)... [2026-10-21 18:40:49][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 18:41:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2384 [2026-10-21 18:41:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2323 [2026-10-21 18:41:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4721 [2026-10-21 18:41:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0294 [2026-10-21 18:41:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5895 [2026-10-21 18:41:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6683 [2026-10-21 18:41:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6644 [2026-10-21 18:41:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5307 [2026-10-21 18:41:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2052 [2026-10-21 18:41:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8711 [2026-10-21 18:41:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1182 [2026-10-21 18:41:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8177 [2026-10-21 18:42:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3437 [2026-10-21 18:42:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7020 [2026-10-21 18:42:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8514 [2026-10-21 18:42:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0803 [2026-10-21 18:42:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7911 [2026-10-21 18:42:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5348 [2026-10-21 18:42:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7675 [2026-10-21 18:42:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.2966 [2026-10-21 18:42:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0939 [2026-10-21 18:42:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2225 [2026-10-21 18:42:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9641 [2026-10-21 18:42:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9543 [2026-10-21 18:42:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7593 [2026-10-21 18:42:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0483 [2026-10-21 18:42:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3076 [2026-10-21 18:42:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6585 [2026-10-21 18:43:07][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,605 | MFU: 1.8% [2026-10-21 18:43:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6251 [2026-10-21 18:43:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4785 [2026-10-21 18:43:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6881 [2026-10-21 18:43:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8951 [2026-10-21 18:43:12][INFO][training.pretrain_engine] [Ph2|Step 299] loss=5.0430 | ppl=154.94 | aux=0.02506 | lr=2.99e-04 | gnorm=0.473 | tok/s=29,386 | MFU=7.1% | step=142733ms | tokens=1.116B | mem=4.2/32.7GB | cost=$24.50/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 05:57 (131.2h) | Phase3: SKIPPED | SFT: Oct 27 10:41 (136.0h) | DPO (Final): Oct 27 11:15 (136.5h) [2026-10-21 18:43:12][INFO][training.pretrain_engine] Starting global step 300 (accumulating 32 micro-batches)... [2026-10-21 18:43:12][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 18:43:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1750 [2026-10-21 18:43:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5511 [2026-10-21 18:43:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6482 [2026-10-21 18:43:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1772 [2026-10-21 18:43:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3872 [2026-10-21 18:43:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1685 [2026-10-21 18:43:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3245 [2026-10-21 18:43:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2774 [2026-10-21 18:44:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6693 [2026-10-21 18:44:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5431 [2026-10-21 18:44:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4895 [2026-10-21 18:44:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3788 [2026-10-21 18:44:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3518 [2026-10-21 18:44:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8938 [2026-10-21 18:44:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7193 [2026-10-21 18:44:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6070 [2026-10-21 18:44:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0773 [2026-10-21 18:44:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9546 [2026-10-21 18:44:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4484 [2026-10-21 18:44:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4432 [2026-10-21 18:44:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0293 [2026-10-21 18:44:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0426 [2026-10-21 18:44:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0318 [2026-10-21 18:44:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8846 [2026-10-21 18:45:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6729 [2026-10-21 18:45:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7559 [2026-10-21 18:45:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6719 [2026-10-21 18:45:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2652 [2026-10-21 18:45:30][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,612 | MFU: 1.8% [2026-10-21 18:45:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2250 [2026-10-21 18:45:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5576 [2026-10-21 18:45:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2876 [2026-10-21 18:45:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3876 [2026-10-21 18:45:35][INFO][training.pretrain_engine] [Ph2|Step 300] loss=5.1011 | ppl=164.19 | aux=0.02506 | lr=2.99e-04 | gnorm=0.496 | tok/s=29,385 | MFU=7.1% | step=142735ms | tokens=1.117B | mem=4.2/32.7GB | cost=$24.58/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 05:59 (131.2h) | Phase3: SKIPPED | SFT: Oct 27 10:43 (136.0h) | DPO (Final): Oct 27 11:17 (136.5h) [2026-10-21 18:45:35][INFO][training.pretrain_engine] Starting global step 301 (accumulating 32 micro-batches)... [2026-10-21 18:45:35][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 18:45:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1533 [2026-10-21 18:45:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3416 [2026-10-21 18:45:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2568 [2026-10-21 18:45:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2979 [2026-10-21 18:46:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5573 [2026-10-21 18:46:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5066 [2026-10-21 18:46:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5995 [2026-10-21 18:46:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1619 [2026-10-21 18:46:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8050 [2026-10-21 18:46:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1513 [2026-10-21 18:46:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4814 [2026-10-21 18:46:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9825 [2026-10-21 18:46:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0950 [2026-10-21 18:46:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1235 [2026-10-21 18:46:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3646 [2026-10-21 18:46:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0923 [2026-10-21 18:47:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8993 [2026-10-21 18:47:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2718 [2026-10-21 18:47:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8504 [2026-10-21 18:47:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3497 [2026-10-21 18:47:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0012 [2026-10-21 18:47:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9226 [2026-10-21 18:47:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9264 [2026-10-21 18:47:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0156 [2026-10-21 18:47:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3377 [2026-10-21 18:47:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1078 [2026-10-21 18:47:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0174 [2026-10-21 18:47:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.1095 [2026-10-21 18:47:53][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,584 | MFU: 1.8% [2026-10-21 18:47:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8235 [2026-10-21 18:47:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8827 [2026-10-21 18:47:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0935 [2026-10-21 18:47:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3812 [2026-10-21 18:47:58][INFO][training.pretrain_engine] [Ph2|Step 301] loss=5.0360 | ppl=153.85 | aux=0.02506 | lr=2.99e-04 | gnorm=0.473 | tok/s=29,385 | MFU=7.1% | step=142734ms | tokens=1.118B | mem=4.2/32.7GB | cost=$24.67/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 06:01 (131.2h) | Phase3: SKIPPED | SFT: Oct 27 10:45 (136.0h) | DPO (Final): Oct 27 11:19 (136.5h) [2026-10-21 18:47:58][INFO][training.pretrain_engine] Starting global step 302 (accumulating 32 micro-batches)... [2026-10-21 18:47:58][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 18:48:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8958 [2026-10-21 18:48:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3370 [2026-10-21 18:48:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9298 [2026-10-21 18:48:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0420 [2026-10-21 18:48:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1971 [2026-10-21 18:48:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1487 [2026-10-21 18:48:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2658 [2026-10-21 18:48:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0436 [2026-10-21 18:48:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8209 [2026-10-21 18:48:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8759 [2026-10-21 18:48:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2866 [2026-10-21 18:48:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9204 [2026-10-21 18:49:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3000 [2026-10-21 18:49:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3066 [2026-10-21 18:49:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8661 [2026-10-21 18:49:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1760 [2026-10-21 18:49:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3979 [2026-10-21 18:49:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4322 [2026-10-21 18:49:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7188 [2026-10-21 18:49:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3687 [2026-10-21 18:49:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5492 [2026-10-21 18:49:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7179 [2026-10-21 18:49:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9991 [2026-10-21 18:49:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0505 [2026-10-21 18:50:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4601 [2026-10-21 18:50:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8261 [2026-10-21 18:50:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1961 [2026-10-21 18:50:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2108 [2026-10-21 18:50:16][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,578 | MFU: 1.8% [2026-10-21 18:50:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5769 [2026-10-21 18:50:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8682 [2026-10-21 18:50:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6636 [2026-10-21 18:50:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7035 [2026-10-21 18:50:21][INFO][training.pretrain_engine] [Ph2|Step 302] loss=5.1499 | ppl=172.41 | aux=0.02506 | lr=2.99e-04 | gnorm=0.543 | tok/s=29,385 | MFU=7.1% | step=142734ms | tokens=1.119B | mem=4.2/32.7GB | cost=$24.75/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 06:03 (131.2h) | Phase3: SKIPPED | SFT: Oct 27 10:46 (135.9h) | DPO (Final): Oct 27 11:21 (136.5h) [2026-10-21 18:50:21][INFO][training.pretrain_engine] Starting global step 303 (accumulating 32 micro-batches)... [2026-10-21 18:50:21][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 18:50:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2503 [2026-10-21 18:50:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8719 [2026-10-21 18:50:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4295 [2026-10-21 18:50:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8314 [2026-10-21 18:50:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2809 [2026-10-21 18:50:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7550 [2026-10-21 18:50:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2874 [2026-10-21 18:50:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3393 [2026-10-21 18:51:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8218 [2026-10-21 18:51:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4176 [2026-10-21 18:51:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5548 [2026-10-21 18:51:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0239 [2026-10-21 18:51:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0540 [2026-10-21 18:51:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0328 [2026-10-21 18:51:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2442 [2026-10-21 18:51:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2835 [2026-10-21 18:51:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9079 [2026-10-21 18:51:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7894 [2026-10-21 18:51:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3723 [2026-10-21 18:51:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9777 [2026-10-21 18:52:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1992 [2026-10-21 18:52:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0192 [2026-10-21 18:52:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3924 [2026-10-21 18:52:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5967 [2026-10-21 18:52:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7775 [2026-10-21 18:52:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3591 [2026-10-21 18:52:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3805 [2026-10-21 18:52:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2743 [2026-10-21 18:52:38][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,611 | MFU: 1.8% [2026-10-21 18:52:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4399 [2026-10-21 18:52:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3332 [2026-10-21 18:52:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4327 [2026-10-21 18:52:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4218 [2026-10-21 18:52:44][INFO][training.pretrain_engine] [Ph2|Step 303] loss=5.2214 | ppl=185.20 | aux=0.02506 | lr=2.99e-04 | gnorm=0.494 | tok/s=29,385 | MFU=7.1% | step=142737ms | tokens=1.120B | mem=4.2/32.7GB | cost=$24.83/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 06:05 (131.2h) | Phase3: SKIPPED | SFT: Oct 27 10:48 (135.9h) | DPO (Final): Oct 27 11:22 (136.5h) [2026-10-21 18:52:44][INFO][training.pretrain_engine] Starting global step 304 (accumulating 32 micro-batches)... [2026-10-21 18:52:44][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 18:53:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3891 [2026-10-21 18:53:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1399 [2026-10-21 18:53:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1531 [2026-10-21 18:53:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3991 [2026-10-21 18:53:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1809 [2026-10-21 18:53:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9209 [2026-10-21 18:53:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1725 [2026-10-21 18:53:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0526 [2026-10-21 18:53:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8897 [2026-10-21 18:53:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2260 [2026-10-21 18:53:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6613 [2026-10-21 18:53:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6304 [2026-10-21 18:53:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0896 [2026-10-21 18:53:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2505 [2026-10-21 18:53:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5181 [2026-10-21 18:53:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2606 [2026-10-21 18:54:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7535 [2026-10-21 18:54:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5086 [2026-10-21 18:54:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2427 [2026-10-21 18:54:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6997 [2026-10-21 18:54:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1567 [2026-10-21 18:54:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8631 [2026-10-21 18:54:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1829 [2026-10-21 18:54:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9323 [2026-10-21 18:54:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0196 [2026-10-21 18:54:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2603 [2026-10-21 18:54:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8230 [2026-10-21 18:54:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3409 [2026-10-21 18:55:01][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,606 | MFU: 1.8% [2026-10-21 18:55:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9302 [2026-10-21 18:55:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0228 [2026-10-21 18:55:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9750 [2026-10-21 18:55:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9724 [2026-10-21 18:55:07][INFO][training.pretrain_engine] [Ph2|Step 304] loss=5.0552 | ppl=156.84 | aux=0.02506 | lr=2.99e-04 | gnorm=0.543 | tok/s=29,383 | MFU=7.1% | step=142746ms | tokens=1.121B | mem=4.2/32.7GB | cost=$24.91/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 06:07 (131.2h) | Phase3: SKIPPED | SFT: Oct 27 10:51 (135.9h) | DPO (Final): Oct 27 11:25 (136.5h) [2026-10-21 18:55:07][INFO][training.pretrain_engine] Starting global step 305 (accumulating 32 micro-batches)... [2026-10-21 18:55:07][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 18:55:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7951 [2026-10-21 18:55:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9139 [2026-10-21 18:55:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8770 [2026-10-21 18:55:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7785 [2026-10-21 18:55:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1201 [2026-10-21 18:55:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5809 [2026-10-21 18:55:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7890 [2026-10-21 18:55:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1634 [2026-10-21 18:56:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0575 [2026-10-21 18:56:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1443 [2026-10-21 18:56:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7752 [2026-10-21 18:56:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5429 [2026-10-21 18:56:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4175 [2026-10-21 18:56:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3606 [2026-10-21 18:56:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4389 [2026-10-21 18:56:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6269 [2026-10-21 18:56:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7037 [2026-10-21 18:56:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8297 [2026-10-21 18:56:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8798 [2026-10-21 18:56:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2872 [2026-10-21 18:56:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4847 [2026-10-21 18:56:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3190 [2026-10-21 18:56:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5245 [2026-10-21 18:56:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2096 [2026-10-21 18:57:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2274 [2026-10-21 18:57:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1208 [2026-10-21 18:57:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1517 [2026-10-21 18:57:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1400 [2026-10-21 18:57:25][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,600 | MFU: 1.8% [2026-10-21 18:57:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9917 [2026-10-21 18:57:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1617 [2026-10-21 18:57:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3278 [2026-10-21 18:57:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0884 [2026-10-21 18:57:30][INFO][training.pretrain_engine] [Ph2|Step 305] loss=5.1958 | ppl=180.52 | aux=0.02506 | lr=2.99e-04 | gnorm=0.508 | tok/s=29,382 | MFU=7.1% | step=142751ms | tokens=1.122B | mem=4.2/32.7GB | cost=$24.99/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 06:09 (131.2h) | Phase3: SKIPPED | SFT: Oct 27 10:53 (135.9h) | DPO (Final): Oct 27 11:27 (136.5h) [2026-10-21 18:57:30][INFO][training.pretrain_engine] Starting global step 306 (accumulating 32 micro-batches)... [2026-10-21 18:57:30][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 18:57:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0982 [2026-10-21 18:57:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6111 [2026-10-21 18:57:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4376 [2026-10-21 18:57:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3681 [2026-10-21 18:58:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1432 [2026-10-21 18:58:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9538 [2026-10-21 18:58:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4875 [2026-10-21 18:58:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0055 [2026-10-21 18:58:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9886 [2026-10-21 18:58:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8631 [2026-10-21 18:58:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7728 [2026-10-21 18:58:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6489 [2026-10-21 18:58:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6983 [2026-10-21 18:58:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0431 [2026-10-21 18:58:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1939 [2026-10-21 18:58:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2620 [2026-10-21 18:58:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9926 [2026-10-21 18:58:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3353 [2026-10-21 18:58:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4927 [2026-10-21 18:59:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9893 [2026-10-21 18:59:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4069 [2026-10-21 18:59:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8489 [2026-10-21 18:59:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2343 [2026-10-21 18:59:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2306 [2026-10-21 18:59:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2026 [2026-10-21 18:59:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0704 [2026-10-21 18:59:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7690 [2026-10-21 18:59:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7329 [2026-10-21 18:59:48][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,595 | MFU: 1.8% [2026-10-21 18:59:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7917 [2026-10-21 18:59:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0475 [2026-10-21 18:59:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9827 [2026-10-21 18:59:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0977 [2026-10-21 18:59:54][INFO][training.pretrain_engine] [Ph2|Step 306] loss=5.0024 | ppl=148.77 | aux=0.02506 | lr=2.99e-04 | gnorm=0.535 | tok/s=29,381 | MFU=7.1% | step=142758ms | tokens=1.123B | mem=4.2/32.7GB | cost=$25.08/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 06:11 (131.2h) | Phase3: SKIPPED | SFT: Oct 27 10:55 (135.9h) | DPO (Final): Oct 27 11:29 (136.5h) [2026-10-21 18:59:57][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step306 | Tokens: 1.123B | Spent: $25.08 | Remaining: $974.92 [2026-10-21 18:59:57][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step306 [2026-10-21 18:59:57][INFO][training.pretrain_engine] Starting global step 307 (accumulating 32 micro-batches)... [2026-10-21 18:59:57][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 18:59:57][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 18:59:57][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 18:59:57][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 18:59:57][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step306 to Phase-Technologies/forge-500m-pretrain (path: phase2_step306)... [2026-10-21 18:59:57][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 19:00:09][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 19:00:09][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step306 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 19:00:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0847 [2026-10-21 19:00:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0701 [2026-10-21 19:00:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2256 [2026-10-21 19:00:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0939 [2026-10-21 19:00:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0884 [2026-10-21 19:00:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2213 [2026-10-21 19:00:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9704 [2026-10-21 19:00:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2216 [2026-10-21 19:00:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9427 [2026-10-21 19:00:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2210 [2026-10-21 19:00:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2490 [2026-10-21 19:00:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1414 [2026-10-21 19:01:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0289 [2026-10-21 19:01:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1835 [2026-10-21 19:01:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6002 [2026-10-21 19:01:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6605 [2026-10-21 19:01:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4816 [2026-10-21 19:01:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5102 [2026-10-21 19:01:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4589 [2026-10-21 19:01:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5451 [2026-10-21 19:01:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3100 [2026-10-21 19:01:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3267 [2026-10-21 19:01:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2903 [2026-10-21 19:01:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2443 [2026-10-21 19:01:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8357 [2026-10-21 19:01:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9473 [2026-10-21 19:01:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3106 [2026-10-21 19:02:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9688 [2026-10-21 19:02:15][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,612 | MFU: 1.8% [2026-10-21 19:02:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3531 [2026-10-21 19:02:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7943 [2026-10-21 19:02:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0413 [2026-10-21 19:02:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9481 [2026-10-21 19:02:19][INFO][training.pretrain_engine] [Ph2|Step 307] loss=5.1123 | ppl=166.06 | aux=0.02505 | lr=2.99e-04 | gnorm=0.562 | tok/s=29,382 | MFU=7.1% | step=142751ms | tokens=1.124B | mem=4.2/32.7GB | cost=$25.16/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 06:13 (131.2h) | Phase3: SKIPPED | SFT: Oct 27 10:56 (135.9h) | DPO (Final): Oct 27 11:31 (136.5h) [2026-10-21 19:02:19][INFO][training.pretrain_engine] Starting global step 308 (accumulating 32 micro-batches)... [2026-10-21 19:02:19][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 19:02:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2120 [2026-10-21 19:02:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7569 [2026-10-21 19:02:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2488 [2026-10-21 19:02:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1037 [2026-10-21 19:02:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5263 [2026-10-21 19:02:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5774 [2026-10-21 19:02:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5990 [2026-10-21 19:02:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8682 [2026-10-21 19:03:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2901 [2026-10-21 19:03:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0341 [2026-10-21 19:03:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2697 [2026-10-21 19:03:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5750 [2026-10-21 19:03:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3937 [2026-10-21 19:03:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5677 [2026-10-21 19:03:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6735 [2026-10-21 19:03:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6291 [2026-10-21 19:03:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2907 [2026-10-21 19:03:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1204 [2026-10-21 19:03:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1751 [2026-10-21 19:03:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0364 [2026-10-21 19:04:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8191 [2026-10-21 19:04:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9174 [2026-10-21 19:04:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2710 [2026-10-21 19:04:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2754 [2026-10-21 19:04:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0550 [2026-10-21 19:04:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3951 [2026-10-21 19:04:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8648 [2026-10-21 19:04:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4560 [2026-10-21 19:04:37][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,598 | MFU: 1.8% [2026-10-21 19:04:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4267 [2026-10-21 19:04:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3215 [2026-10-21 19:04:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5860 [2026-10-21 19:04:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3837 [2026-10-21 19:04:42][INFO][training.pretrain_engine] [Ph2|Step 308] loss=5.0528 | ppl=156.46 | aux=0.02506 | lr=2.99e-04 | gnorm=0.562 | tok/s=29,381 | MFU=7.1% | step=142757ms | tokens=1.125B | mem=4.2/32.7GB | cost=$25.24/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 06:15 (131.2h) | Phase3: SKIPPED | SFT: Oct 27 10:59 (135.9h) | DPO (Final): Oct 27 11:33 (136.5h) [2026-10-21 19:04:42][INFO][training.pretrain_engine] Starting global step 309 (accumulating 32 micro-batches)... [2026-10-21 19:04:42][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 19:05:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6942 [2026-10-21 19:05:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0184 [2026-10-21 19:05:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9806 [2026-10-21 19:05:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0338 [2026-10-21 19:05:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2007 [2026-10-21 19:05:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1479 [2026-10-21 19:05:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1607 [2026-10-21 19:05:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4690 [2026-10-21 19:05:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5645 [2026-10-21 19:05:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2799 [2026-10-21 19:05:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0625 [2026-10-21 19:05:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2089 [2026-10-21 19:05:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2009 [2026-10-21 19:05:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2320 [2026-10-21 19:05:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1037 [2026-10-21 19:05:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3926 [2026-10-21 19:06:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2601 [2026-10-21 19:06:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8866 [2026-10-21 19:06:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4901 [2026-10-21 19:06:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9159 [2026-10-21 19:06:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8534 [2026-10-21 19:06:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1777 [2026-10-21 19:06:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1392 [2026-10-21 19:06:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0177 [2026-10-21 19:06:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4378 [2026-10-21 19:06:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6168 [2026-10-21 19:06:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6730 [2026-10-21 19:06:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4028 [2026-10-21 19:07:00][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,581 | MFU: 1.8% [2026-10-21 19:07:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3787 [2026-10-21 19:07:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2054 [2026-10-21 19:07:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4411 [2026-10-21 19:07:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7306 [2026-10-21 19:07:05][INFO][training.pretrain_engine] [Ph2|Step 309] loss=5.1084 | ppl=165.41 | aux=0.02507 | lr=2.99e-04 | gnorm=0.531 | tok/s=29,380 | MFU=7.1% | step=142761ms | tokens=1.126B | mem=4.2/32.7GB | cost=$25.32/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 06:17 (131.2h) | Phase3: SKIPPED | SFT: Oct 27 11:01 (135.9h) | DPO (Final): Oct 27 11:35 (136.5h) [2026-10-21 19:07:05][INFO][training.pretrain_engine] Starting global step 310 (accumulating 32 micro-batches)... [2026-10-21 19:07:05][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 19:07:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0395 [2026-10-21 19:07:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1969 [2026-10-21 19:07:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1469 [2026-10-21 19:07:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3634 [2026-10-21 19:07:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3886 [2026-10-21 19:07:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1132 [2026-10-21 19:07:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6772 [2026-10-21 19:07:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0465 [2026-10-21 19:07:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0071 [2026-10-21 19:07:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8728 [2026-10-21 19:07:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0260 [2026-10-21 19:07:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0969 [2026-10-21 19:08:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5121 [2026-10-21 19:08:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7785 [2026-10-21 19:08:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2950 [2026-10-21 19:08:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7471 [2026-10-21 19:08:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7793 [2026-10-21 19:08:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6392 [2026-10-21 19:08:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5219 [2026-10-21 19:08:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.0453 [2026-10-21 19:08:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2203 [2026-10-21 19:08:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6594 [2026-10-21 19:08:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7116 [2026-10-21 19:08:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5311 [2026-10-21 19:09:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8228 [2026-10-21 19:09:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9924 [2026-10-21 19:09:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3432 [2026-10-21 19:09:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2539 [2026-10-21 19:09:23][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,602 | MFU: 1.8% [2026-10-21 19:09:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8657 [2026-10-21 19:09:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3748 [2026-10-21 19:09:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2530 [2026-10-21 19:09:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9702 [2026-10-21 19:09:28][INFO][training.pretrain_engine] [Ph2|Step 310] loss=5.0588 | ppl=157.40 | aux=0.02506 | lr=2.99e-04 | gnorm=0.672 | tok/s=29,379 | MFU=7.1% | step=142763ms | tokens=1.127B | mem=4.2/32.7GB | cost=$25.40/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 06:19 (131.2h) | Phase3: SKIPPED | SFT: Oct 27 11:02 (135.9h) | DPO (Final): Oct 27 11:37 (136.5h) [2026-10-21 19:09:28][INFO][training.pretrain_engine] Starting global step 311 (accumulating 32 micro-batches)... [2026-10-21 19:09:28][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 19:09:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2672 [2026-10-21 19:09:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0277 [2026-10-21 19:09:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3718 [2026-10-21 19:09:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5050 [2026-10-21 19:10:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0939 [2026-10-21 19:10:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9412 [2026-10-21 19:10:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0674 [2026-10-21 19:10:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0268 [2026-10-21 19:10:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7095 [2026-10-21 19:10:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8136 [2026-10-21 19:10:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7938 [2026-10-21 19:10:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6105 [2026-10-21 19:10:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7722 [2026-10-21 19:10:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9583 [2026-10-21 19:10:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3345 [2026-10-21 19:10:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4209 [2026-10-21 19:10:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6676 [2026-10-21 19:10:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0016 [2026-10-21 19:10:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2012 [2026-10-21 19:10:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0074 [2026-10-21 19:11:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3816 [2026-10-21 19:11:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6420 [2026-10-21 19:11:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1996 [2026-10-21 19:11:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9074 [2026-10-21 19:11:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8570 [2026-10-21 19:11:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2772 [2026-10-21 19:11:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9951 [2026-10-21 19:11:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3316 [2026-10-21 19:11:46][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,600 | MFU: 1.8% [2026-10-21 19:11:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1762 [2026-10-21 19:11:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5352 [2026-10-21 19:11:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0348 [2026-10-21 19:11:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3871 [2026-10-21 19:11:52][INFO][training.pretrain_engine] [Ph2|Step 311] loss=4.9845 | ppl=146.13 | aux=0.02506 | lr=2.99e-04 | gnorm=0.844 | tok/s=29,377 | MFU=7.1% | step=142773ms | tokens=1.128B | mem=4.2/32.7GB | cost=$25.49/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 06:21 (131.2h) | Phase3: SKIPPED | SFT: Oct 27 11:05 (135.9h) | DPO (Final): Oct 27 11:39 (136.5h) [2026-10-21 19:11:52][INFO][training.pretrain_engine] Starting global step 312 (accumulating 32 micro-batches)... [2026-10-21 19:11:52][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 19:12:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7023 [2026-10-21 19:12:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3118 [2026-10-21 19:12:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6836 [2026-10-21 19:12:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7823 [2026-10-21 19:12:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4748 [2026-10-21 19:12:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2989 [2026-10-21 19:12:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9319 [2026-10-21 19:12:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9635 [2026-10-21 19:12:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5983 [2026-10-21 19:12:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8496 [2026-10-21 19:12:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4855 [2026-10-21 19:12:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2533 [2026-10-21 19:13:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9900 [2026-10-21 19:13:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8687 [2026-10-21 19:13:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0162 [2026-10-21 19:13:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5093 [2026-10-21 19:13:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9768 [2026-10-21 19:13:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4320 [2026-10-21 19:13:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8281 [2026-10-21 19:13:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8467 [2026-10-21 19:13:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4580 [2026-10-21 19:13:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3471 [2026-10-21 19:13:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2278 [2026-10-21 19:13:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4034 [2026-10-21 19:13:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3522 [2026-10-21 19:13:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.1517 [2026-10-21 19:13:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7549 [2026-10-21 19:13:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1029 [2026-10-21 19:14:10][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,592 | MFU: 1.8% [2026-10-21 19:14:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0509 [2026-10-21 19:14:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1437 [2026-10-21 19:14:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9712 [2026-10-21 19:14:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7013 [2026-10-21 19:14:15][INFO][training.pretrain_engine] [Ph2|Step 312] loss=5.0348 | ppl=153.67 | aux=0.02507 | lr=2.99e-04 | gnorm=0.957 | tok/s=29,377 | MFU=7.1% | step=142776ms | tokens=1.129B | mem=4.2/32.7GB | cost=$25.57/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 06:23 (131.2h) | Phase3: SKIPPED | SFT: Oct 27 11:07 (135.9h) | DPO (Final): Oct 27 11:41 (136.5h) [2026-10-21 19:14:15][INFO][training.pretrain_engine] Starting global step 313 (accumulating 32 micro-batches)... [2026-10-21 19:14:15][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 19:14:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7574 [2026-10-21 19:14:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6092 [2026-10-21 19:14:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7248 [2026-10-21 19:14:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7301 [2026-10-21 19:14:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3678 [2026-10-21 19:14:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0478 [2026-10-21 19:14:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3285 [2026-10-21 19:14:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8854 [2026-10-21 19:15:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6251 [2026-10-21 19:15:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0575 [2026-10-21 19:15:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1318 [2026-10-21 19:15:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7007 [2026-10-21 19:15:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3234 [2026-10-21 19:15:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6807 [2026-10-21 19:15:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2737 [2026-10-21 19:15:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4873 [2026-10-21 19:15:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2168 [2026-10-21 19:15:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8137 [2026-10-21 19:15:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7335 [2026-10-21 19:15:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1809 [2026-10-21 19:16:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2956 [2026-10-21 19:16:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6063 [2026-10-21 19:16:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5663 [2026-10-21 19:16:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5504 [2026-10-21 19:16:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6498 [2026-10-21 19:16:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2706 [2026-10-21 19:16:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4093 [2026-10-21 19:16:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0929 [2026-10-21 19:16:33][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,591 | MFU: 1.8% [2026-10-21 19:16:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4358 [2026-10-21 19:16:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9253 [2026-10-21 19:16:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4796 [2026-10-21 19:16:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9738 [2026-10-21 19:16:38][INFO][training.pretrain_engine] [Ph2|Step 313] loss=5.1850 | ppl=178.57 | aux=0.02507 | lr=2.99e-04 | gnorm=1.133 | tok/s=29,376 | MFU=7.1% | step=142781ms | tokens=1.130B | mem=4.2/32.7GB | cost=$25.65/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 06:25 (131.2h) | Phase3: SKIPPED | SFT: Oct 27 11:09 (135.9h) | DPO (Final): Oct 27 11:43 (136.4h) [2026-10-21 19:16:38][INFO][training.pretrain_engine] Starting global step 314 (accumulating 32 micro-batches)... [2026-10-21 19:16:38][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 19:16:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6034 [2026-10-21 19:16:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6840 [2026-10-21 19:16:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3957 [2026-10-21 19:16:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2127 [2026-10-21 19:17:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0203 [2026-10-21 19:17:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3845 [2026-10-21 19:17:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8986 [2026-10-21 19:17:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0967 [2026-10-21 19:17:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9299 [2026-10-21 19:17:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0731 [2026-10-21 19:17:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6537 [2026-10-21 19:17:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8409 [2026-10-21 19:17:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9163 [2026-10-21 19:17:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7433 [2026-10-21 19:17:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8754 [2026-10-21 19:17:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5053 [2026-10-21 19:18:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3238 [2026-10-21 19:18:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9771 [2026-10-21 19:18:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6304 [2026-10-21 19:18:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0274 [2026-10-21 19:18:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1378 [2026-10-21 19:18:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4405 [2026-10-21 19:18:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4480 [2026-10-21 19:18:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1629 [2026-10-21 19:18:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8151 [2026-10-21 19:18:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5072 [2026-10-21 19:18:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5521 [2026-10-21 19:18:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0793 [2026-10-21 19:18:56][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,589 | MFU: 1.8% [2026-10-21 19:19:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1290 [2026-10-21 19:19:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6477 [2026-10-21 19:19:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5855 [2026-10-21 19:19:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0610 [2026-10-21 19:19:02][INFO][training.pretrain_engine] [Ph2|Step 314] loss=5.2030 | ppl=181.82 | aux=0.02506 | lr=2.99e-04 | gnorm=0.895 | tok/s=29,375 | MFU=7.1% | step=142784ms | tokens=1.131B | mem=4.2/32.7GB | cost=$25.73/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 06:27 (131.1h) | Phase3: SKIPPED | SFT: Oct 27 11:11 (135.9h) | DPO (Final): Oct 27 11:45 (136.4h) [2026-10-21 19:19:02][INFO][training.pretrain_engine] Starting global step 315 (accumulating 32 micro-batches)... [2026-10-21 19:19:02][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 19:19:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7855 [2026-10-21 19:19:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2894 [2026-10-21 19:19:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2841 [2026-10-21 19:19:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7651 [2026-10-21 19:19:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9929 [2026-10-21 19:19:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7485 [2026-10-21 19:19:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8152 [2026-10-21 19:19:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8030 [2026-10-21 19:19:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1031 [2026-10-21 19:19:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0217 [2026-10-21 19:19:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8669 [2026-10-21 19:19:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4021 [2026-10-21 19:20:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1338 [2026-10-21 19:20:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2628 [2026-10-21 19:20:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2996 [2026-10-21 19:20:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8232 [2026-10-21 19:20:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5343 [2026-10-21 19:20:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1004 [2026-10-21 19:20:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5244 [2026-10-21 19:20:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7676 [2026-10-21 19:20:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3276 [2026-10-21 19:20:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8749 [2026-10-21 19:20:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5384 [2026-10-21 19:20:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9400 [2026-10-21 19:21:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1006 [2026-10-21 19:21:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3714 [2026-10-21 19:21:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9036 [2026-10-21 19:21:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2492 [2026-10-21 19:21:19][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,603 | MFU: 1.8% [2026-10-21 19:21:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9474 [2026-10-21 19:21:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3992 [2026-10-21 19:21:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1157 [2026-10-21 19:21:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2566 [2026-10-21 19:21:25][INFO][training.pretrain_engine] [Ph2|Step 315] loss=5.0341 | ppl=153.56 | aux=0.02507 | lr=2.99e-04 | gnorm=0.742 | tok/s=29,374 | MFU=7.1% | step=142788ms | tokens=1.132B | mem=4.2/32.7GB | cost=$25.81/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 06:29 (131.1h) | Phase3: SKIPPED | SFT: Oct 27 11:13 (135.9h) | DPO (Final): Oct 27 11:47 (136.4h) [2026-10-21 19:21:28][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step315 | Tokens: 1.132B | Spent: $25.82 | Remaining: $974.18 [2026-10-21 19:21:28][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step315 [2026-10-21 19:21:28][INFO][training.pretrain_engine] Starting global step 316 (accumulating 32 micro-batches)... [2026-10-21 19:21:28][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 19:21:28][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 19:21:28][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 19:21:28][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 19:21:28][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step315 to Phase-Technologies/forge-500m-pretrain (path: phase2_step315)... [2026-10-21 19:21:28][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 19:21:38][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 19:21:38][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step315 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 19:21:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5219 [2026-10-21 19:21:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7208 [2026-10-21 19:21:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9883 [2026-10-21 19:21:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0335 [2026-10-21 19:22:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2444 [2026-10-21 19:22:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4584 [2026-10-21 19:22:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2636 [2026-10-21 19:22:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9532 [2026-10-21 19:22:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2121 [2026-10-21 19:22:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2224 [2026-10-21 19:22:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1568 [2026-10-21 19:22:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2442 [2026-10-21 19:22:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4231 [2026-10-21 19:22:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3513 [2026-10-21 19:22:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5421 [2026-10-21 19:22:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2364 [2026-10-21 19:22:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3306 [2026-10-21 19:22:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1830 [2026-10-21 19:22:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8076 [2026-10-21 19:22:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2380 [2026-10-21 19:23:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1971 [2026-10-21 19:23:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0166 [2026-10-21 19:23:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6051 [2026-10-21 19:23:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3326 [2026-10-21 19:23:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5309 [2026-10-21 19:23:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3715 [2026-10-21 19:23:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8290 [2026-10-21 19:23:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4218 [2026-10-21 19:23:46][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,595 | MFU: 1.8% [2026-10-21 19:23:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5755 [2026-10-21 19:23:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6313 [2026-10-21 19:23:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2076 [2026-10-21 19:23:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6334 [2026-10-21 19:23:50][INFO][training.pretrain_engine] [Ph2|Step 316] loss=5.1386 | ppl=170.48 | aux=0.02507 | lr=2.99e-04 | gnorm=0.684 | tok/s=29,375 | MFU=7.1% | step=142785ms | tokens=1.134B | mem=4.2/32.7GB | cost=$25.90/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 06:31 (131.1h) | Phase3: SKIPPED | SFT: Oct 27 11:15 (135.9h) | DPO (Final): Oct 27 11:49 (136.4h) [2026-10-21 19:23:50][INFO][training.pretrain_engine] Starting global step 317 (accumulating 32 micro-batches)... [2026-10-21 19:23:50][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 19:24:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1759 [2026-10-21 19:24:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0421 [2026-10-21 19:24:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6892 [2026-10-21 19:24:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0457 [2026-10-21 19:24:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5276 [2026-10-21 19:24:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6856 [2026-10-21 19:24:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3417 [2026-10-21 19:24:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5074 [2026-10-21 19:24:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0361 [2026-10-21 19:24:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2824 [2026-10-21 19:24:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1417 [2026-10-21 19:24:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1094 [2026-10-21 19:25:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1552 [2026-10-21 19:25:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8545 [2026-10-21 19:25:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0748 [2026-10-21 19:25:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2070 [2026-10-21 19:25:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0510 [2026-10-21 19:25:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5793 [2026-10-21 19:25:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6130 [2026-10-21 19:25:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6677 [2026-10-21 19:25:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5337 [2026-10-21 19:25:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5060 [2026-10-21 19:25:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1745 [2026-10-21 19:25:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3980 [2026-10-21 19:25:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9468 [2026-10-21 19:25:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1948 [2026-10-21 19:25:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9287 [2026-10-21 19:25:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1019 [2026-10-21 19:26:08][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,598 | MFU: 1.8% [2026-10-21 19:26:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1351 [2026-10-21 19:26:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8552 [2026-10-21 19:26:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8159 [2026-10-21 19:26:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0392 [2026-10-21 19:26:14][INFO][training.pretrain_engine] [Ph2|Step 317] loss=5.0588 | ppl=157.40 | aux=0.02507 | lr=2.99e-04 | gnorm=0.992 | tok/s=29,372 | MFU=7.0% | step=142798ms | tokens=1.135B | mem=4.2/32.7GB | cost=$25.98/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 06:33 (131.1h) | Phase3: SKIPPED | SFT: Oct 27 11:17 (135.9h) | DPO (Final): Oct 27 11:51 (136.4h) [2026-10-21 19:26:14][INFO][training.pretrain_engine] Starting global step 318 (accumulating 32 micro-batches)... [2026-10-21 19:26:14][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 19:26:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3123 [2026-10-21 19:26:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5137 [2026-10-21 19:26:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4076 [2026-10-21 19:26:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8091 [2026-10-21 19:26:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1137 [2026-10-21 19:26:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3069 [2026-10-21 19:26:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5791 [2026-10-21 19:26:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0459 [2026-10-21 19:27:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3685 [2026-10-21 19:27:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6889 [2026-10-21 19:27:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4212 [2026-10-21 19:27:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6033 [2026-10-21 19:27:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1102 [2026-10-21 19:27:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7521 [2026-10-21 19:27:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9915 [2026-10-21 19:27:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8940 [2026-10-21 19:27:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1287 [2026-10-21 19:27:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0482 [2026-10-21 19:27:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9054 [2026-10-21 19:27:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4691 [2026-10-21 19:28:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0602 [2026-10-21 19:28:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7834 [2026-10-21 19:28:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1583 [2026-10-21 19:28:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0155 [2026-10-21 19:28:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3395 [2026-10-21 19:28:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9197 [2026-10-21 19:28:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2119 [2026-10-21 19:28:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1644 [2026-10-21 19:28:31][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,615 | MFU: 1.8% [2026-10-21 19:28:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6525 [2026-10-21 19:28:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0572 [2026-10-21 19:28:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0746 [2026-10-21 19:28:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8088 [2026-10-21 19:28:37][INFO][training.pretrain_engine] [Ph2|Step 318] loss=5.1107 | ppl=165.78 | aux=0.02506 | lr=2.99e-04 | gnorm=0.684 | tok/s=29,371 | MFU=7.0% | step=142803ms | tokens=1.136B | mem=4.2/32.7GB | cost=$26.06/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 06:35 (131.1h) | Phase3: SKIPPED | SFT: Oct 27 11:19 (135.9h) | DPO (Final): Oct 27 11:53 (136.4h) [2026-10-21 19:28:37][INFO][training.pretrain_engine] Starting global step 319 (accumulating 32 micro-batches)... [2026-10-21 19:28:37][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 19:28:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8729 [2026-10-21 19:28:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2043 [2026-10-21 19:28:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3273 [2026-10-21 19:28:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2034 [2026-10-21 19:29:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3181 [2026-10-21 19:29:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5380 [2026-10-21 19:29:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2822 [2026-10-21 19:29:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4716 [2026-10-21 19:29:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1977 [2026-10-21 19:29:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0174 [2026-10-21 19:29:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3616 [2026-10-21 19:29:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2963 [2026-10-21 19:29:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8577 [2026-10-21 19:29:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1164 [2026-10-21 19:29:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0924 [2026-10-21 19:29:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8203 [2026-10-21 19:30:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2176 [2026-10-21 19:30:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1734 [2026-10-21 19:30:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2167 [2026-10-21 19:30:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0574 [2026-10-21 19:30:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9849 [2026-10-21 19:30:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0345 [2026-10-21 19:30:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6917 [2026-10-21 19:30:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0461 [2026-10-21 19:30:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2246 [2026-10-21 19:30:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1670 [2026-10-21 19:30:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2895 [2026-10-21 19:30:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3847 [2026-10-21 19:30:55][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,608 | MFU: 1.8% [2026-10-21 19:31:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3278 [2026-10-21 19:31:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0298 [2026-10-21 19:31:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0745 [2026-10-21 19:31:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7458 [2026-10-21 19:31:00][INFO][training.pretrain_engine] [Ph2|Step 319] loss=5.1961 | ppl=180.56 | aux=0.02507 | lr=2.99e-04 | gnorm=0.879 | tok/s=29,371 | MFU=7.0% | step=142806ms | tokens=1.137B | mem=4.2/32.7GB | cost=$26.14/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 06:37 (131.1h) | Phase3: SKIPPED | SFT: Oct 27 11:21 (135.8h) | DPO (Final): Oct 27 11:55 (136.4h) [2026-10-21 19:31:00][INFO][training.pretrain_engine] Starting global step 320 (accumulating 32 micro-batches)... [2026-10-21 19:31:00][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 19:31:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7565 [2026-10-21 19:31:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2535 [2026-10-21 19:31:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8474 [2026-10-21 19:31:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3105 [2026-10-21 19:31:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9235 [2026-10-21 19:31:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5991 [2026-10-21 19:31:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7554 [2026-10-21 19:31:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0915 [2026-10-21 19:31:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7005 [2026-10-21 19:31:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7730 [2026-10-21 19:31:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4676 [2026-10-21 19:31:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5572 [2026-10-21 19:32:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2173 [2026-10-21 19:32:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4605 [2026-10-21 19:32:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5114 [2026-10-21 19:32:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1441 [2026-10-21 19:32:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4514 [2026-10-21 19:32:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4726 [2026-10-21 19:32:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6076 [2026-10-21 19:32:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8570 [2026-10-21 19:32:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.9547 [2026-10-21 19:32:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1383 [2026-10-21 19:32:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0153 [2026-10-21 19:32:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2812 [2026-10-21 19:33:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9296 [2026-10-21 19:33:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7622 [2026-10-21 19:33:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3299 [2026-10-21 19:33:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8583 [2026-10-21 19:33:17][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,616 | MFU: 1.8% [2026-10-21 19:33:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6842 [2026-10-21 19:33:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8712 [2026-10-21 19:33:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6514 [2026-10-21 19:33:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9978 [2026-10-21 19:33:23][INFO][training.pretrain_engine] [Ph2|Step 320] loss=5.0789 | ppl=160.60 | aux=0.02507 | lr=2.99e-04 | gnorm=0.695 | tok/s=29,370 | MFU=7.0% | step=142808ms | tokens=1.138B | mem=4.2/32.7GB | cost=$26.23/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 06:39 (131.1h) | Phase3: SKIPPED | SFT: Oct 27 11:23 (135.8h) | DPO (Final): Oct 27 11:57 (136.4h) [2026-10-21 19:33:23][INFO][training.pretrain_engine] Starting global step 321 (accumulating 32 micro-batches)... [2026-10-21 19:33:23][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 19:33:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1060 [2026-10-21 19:33:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6336 [2026-10-21 19:33:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3155 [2026-10-21 19:33:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5603 [2026-10-21 19:33:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9913 [2026-10-21 19:33:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0900 [2026-10-21 19:33:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8608 [2026-10-21 19:33:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1863 [2026-10-21 19:34:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3769 [2026-10-21 19:34:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9164 [2026-10-21 19:34:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0107 [2026-10-21 19:34:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0617 [2026-10-21 19:34:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0342 [2026-10-21 19:34:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7139 [2026-10-21 19:34:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9608 [2026-10-21 19:34:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6869 [2026-10-21 19:34:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4237 [2026-10-21 19:34:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8255 [2026-10-21 19:34:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0629 [2026-10-21 19:34:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7989 [2026-10-21 19:35:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7725 [2026-10-21 19:35:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1101 [2026-10-21 19:35:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1393 [2026-10-21 19:35:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5408 [2026-10-21 19:35:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8788 [2026-10-21 19:35:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1617 [2026-10-21 19:35:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0560 [2026-10-21 19:35:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8556 [2026-10-21 19:35:40][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,616 | MFU: 1.8% [2026-10-21 19:35:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4958 [2026-10-21 19:35:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0511 [2026-10-21 19:35:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0870 [2026-10-21 19:35:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0154 [2026-10-21 19:35:46][INFO][training.pretrain_engine] [Ph2|Step 321] loss=5.0953 | ppl=163.26 | aux=0.02507 | lr=2.99e-04 | gnorm=0.758 | tok/s=29,369 | MFU=7.0% | step=142812ms | tokens=1.139B | mem=4.2/32.7GB | cost=$26.31/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 06:41 (131.1h) | Phase3: SKIPPED | SFT: Oct 27 11:25 (135.8h) | DPO (Final): Oct 27 11:59 (136.4h) [2026-10-21 19:35:46][INFO][training.pretrain_engine] Starting global step 322 (accumulating 32 micro-batches)... [2026-10-21 19:35:46][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 19:36:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4659 [2026-10-21 19:36:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6903 [2026-10-21 19:36:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9934 [2026-10-21 19:36:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0866 [2026-10-21 19:36:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8996 [2026-10-21 19:36:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0103 [2026-10-21 19:36:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9386 [2026-10-21 19:36:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8806 [2026-10-21 19:36:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2913 [2026-10-21 19:36:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9255 [2026-10-21 19:36:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5292 [2026-10-21 19:36:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1673 [2026-10-21 19:36:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1622 [2026-10-21 19:36:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8222 [2026-10-21 19:36:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1298 [2026-10-21 19:36:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7256 [2026-10-21 19:37:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2176 [2026-10-21 19:37:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4965 [2026-10-21 19:37:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7835 [2026-10-21 19:37:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3700 [2026-10-21 19:37:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6542 [2026-10-21 19:37:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7827 [2026-10-21 19:37:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9728 [2026-10-21 19:37:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9692 [2026-10-21 19:37:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8378 [2026-10-21 19:37:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8128 [2026-10-21 19:37:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6400 [2026-10-21 19:37:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5530 [2026-10-21 19:38:04][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,601 | MFU: 1.8% [2026-10-21 19:38:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3451 [2026-10-21 19:38:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1121 [2026-10-21 19:38:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2120 [2026-10-21 19:38:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0724 [2026-10-21 19:38:09][INFO][training.pretrain_engine] [Ph2|Step 322] loss=5.0606 | ppl=157.68 | aux=0.02507 | lr=2.99e-04 | gnorm=0.719 | tok/s=29,369 | MFU=7.0% | step=142814ms | tokens=1.140B | mem=4.2/32.7GB | cost=$26.39/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 06:43 (131.1h) | Phase3: SKIPPED | SFT: Oct 27 11:27 (135.8h) | DPO (Final): Oct 27 12:01 (136.4h) [2026-10-21 19:38:09][INFO][training.pretrain_engine] Starting global step 323 (accumulating 32 micro-batches)... [2026-10-21 19:38:09][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 19:38:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5469 [2026-10-21 19:38:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5732 [2026-10-21 19:38:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8157 [2026-10-21 19:38:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4131 [2026-10-21 19:38:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.4747 [2026-10-21 19:38:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3291 [2026-10-21 19:38:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5264 [2026-10-21 19:38:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4250 [2026-10-21 19:39:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8826 [2026-10-21 19:39:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9865 [2026-10-21 19:39:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6062 [2026-10-21 19:39:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8672 [2026-10-21 19:39:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2074 [2026-10-21 19:39:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3334 [2026-10-21 19:39:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5939 [2026-10-21 19:39:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3409 [2026-10-21 19:39:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8642 [2026-10-21 19:39:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7259 [2026-10-21 19:39:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6968 [2026-10-21 19:39:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0310 [2026-10-21 19:39:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3372 [2026-10-21 19:39:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5471 [2026-10-21 19:39:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2434 [2026-10-21 19:39:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1546 [2026-10-21 19:40:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1545 [2026-10-21 19:40:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5906 [2026-10-21 19:40:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9456 [2026-10-21 19:40:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0136 [2026-10-21 19:40:27][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,591 | MFU: 1.8% [2026-10-21 19:40:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0473 [2026-10-21 19:40:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5185 [2026-10-21 19:40:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0004 [2026-10-21 19:40:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0850 [2026-10-21 19:40:32][INFO][training.pretrain_engine] [Ph2|Step 323] loss=5.0994 | ppl=163.92 | aux=0.02508 | lr=2.99e-04 | gnorm=0.617 | tok/s=29,368 | MFU=7.0% | step=142817ms | tokens=1.141B | mem=4.2/32.7GB | cost=$26.47/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 06:45 (131.1h) | Phase3: SKIPPED | SFT: Oct 27 11:29 (135.8h) | DPO (Final): Oct 27 12:03 (136.4h) [2026-10-21 19:40:32][INFO][training.pretrain_engine] Starting global step 324 (accumulating 32 micro-batches)... [2026-10-21 19:40:32][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 19:40:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9583 [2026-10-21 19:40:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2795 [2026-10-21 19:40:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2610 [2026-10-21 19:40:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1241 [2026-10-21 19:41:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1926 [2026-10-21 19:41:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6597 [2026-10-21 19:41:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7926 [2026-10-21 19:41:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8120 [2026-10-21 19:41:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4304 [2026-10-21 19:41:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8689 [2026-10-21 19:41:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2276 [2026-10-21 19:41:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4631 [2026-10-21 19:41:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0421 [2026-10-21 19:41:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9947 [2026-10-21 19:41:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7105 [2026-10-21 19:41:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1766 [2026-10-21 19:42:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3164 [2026-10-21 19:42:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6856 [2026-10-21 19:42:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2405 [2026-10-21 19:42:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3915 [2026-10-21 19:42:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2958 [2026-10-21 19:42:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2834 [2026-10-21 19:42:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4718 [2026-10-21 19:42:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5698 [2026-10-21 19:42:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4432 [2026-10-21 19:42:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8954 [2026-10-21 19:42:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3190 [2026-10-21 19:42:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7711 [2026-10-21 19:42:50][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,602 | MFU: 1.8% [2026-10-21 19:42:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3007 [2026-10-21 19:42:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2696 [2026-10-21 19:42:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2517 [2026-10-21 19:42:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0531 [2026-10-21 19:42:56][INFO][training.pretrain_engine] [Ph2|Step 324] loss=5.1095 | ppl=165.60 | aux=0.02506 | lr=2.99e-04 | gnorm=0.613 | tok/s=29,367 | MFU=7.0% | step=142823ms | tokens=1.142B | mem=4.2/32.7GB | cost=$26.55/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 06:47 (131.1h) | Phase3: SKIPPED | SFT: Oct 27 11:31 (135.8h) | DPO (Final): Oct 27 12:05 (136.4h) [2026-10-21 19:42:59][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step324 | Tokens: 1.142B | Spent: $26.56 | Remaining: $973.44 [2026-10-21 19:42:59][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step324 [2026-10-21 19:42:59][INFO][training.pretrain_engine] Starting global step 325 (accumulating 32 micro-batches)... [2026-10-21 19:42:59][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 19:42:59][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 19:42:59][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 19:42:59][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 19:42:59][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step324 to Phase-Technologies/forge-500m-pretrain (path: phase2_step324)... [2026-10-21 19:42:59][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 19:43:09][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 19:43:09][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step324 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 19:43:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3908 [2026-10-21 19:43:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2560 [2026-10-21 19:43:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2543 [2026-10-21 19:43:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3919 [2026-10-21 19:43:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4892 [2026-10-21 19:43:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4127 [2026-10-21 19:43:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9759 [2026-10-21 19:43:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0355 [2026-10-21 19:43:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9506 [2026-10-21 19:43:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1356 [2026-10-21 19:43:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7598 [2026-10-21 19:43:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1371 [2026-10-21 19:44:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6407 [2026-10-21 19:44:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5052 [2026-10-21 19:44:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8532 [2026-10-21 19:44:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9432 [2026-10-21 19:44:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3728 [2026-10-21 19:44:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6634 [2026-10-21 19:44:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4454 [2026-10-21 19:44:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8291 [2026-10-21 19:44:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8047 [2026-10-21 19:44:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7363 [2026-10-21 19:44:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7463 [2026-10-21 19:44:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1034 [2026-10-21 19:45:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0599 [2026-10-21 19:45:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8857 [2026-10-21 19:45:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9122 [2026-10-21 19:45:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0265 [2026-10-21 19:45:17][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,586 | MFU: 1.8% [2026-10-21 19:45:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8986 [2026-10-21 19:45:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0623 [2026-10-21 19:45:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4719 [2026-10-21 19:45:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9682 [2026-10-21 19:45:21][INFO][training.pretrain_engine] [Ph2|Step 325] loss=5.1556 | ppl=173.40 | aux=0.02507 | lr=2.99e-04 | gnorm=0.590 | tok/s=29,369 | MFU=7.0% | step=142812ms | tokens=1.143B | mem=4.2/32.7GB | cost=$26.64/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 06:49 (131.1h) | Phase3: SKIPPED | SFT: Oct 27 11:32 (135.8h) | DPO (Final): Oct 27 12:06 (136.4h) [2026-10-21 19:45:21][INFO][training.pretrain_engine] Starting global step 326 (accumulating 32 micro-batches)... [2026-10-21 19:45:21][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 19:45:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7137 [2026-10-21 19:45:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5522 [2026-10-21 19:45:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2533 [2026-10-21 19:45:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3386 [2026-10-21 19:45:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0326 [2026-10-21 19:45:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8043 [2026-10-21 19:45:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2621 [2026-10-21 19:45:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4173 [2026-10-21 19:46:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7493 [2026-10-21 19:46:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3524 [2026-10-21 19:46:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4472 [2026-10-21 19:46:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2494 [2026-10-21 19:46:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7982 [2026-10-21 19:46:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2720 [2026-10-21 19:46:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3287 [2026-10-21 19:46:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1702 [2026-10-21 19:46:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7412 [2026-10-21 19:46:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6700 [2026-10-21 19:46:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9325 [2026-10-21 19:46:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0700 [2026-10-21 19:47:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9866 [2026-10-21 19:47:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6949 [2026-10-21 19:47:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9418 [2026-10-21 19:47:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3948 [2026-10-21 19:47:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1024 [2026-10-21 19:47:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3167 [2026-10-21 19:47:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9606 [2026-10-21 19:47:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0693 [2026-10-21 19:47:39][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,606 | MFU: 1.8% [2026-10-21 19:47:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8894 [2026-10-21 19:47:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3086 [2026-10-21 19:47:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0669 [2026-10-21 19:47:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3991 [2026-10-21 19:47:45][INFO][training.pretrain_engine] [Ph2|Step 326] loss=5.0914 | ppl=162.62 | aux=0.02506 | lr=2.99e-04 | gnorm=0.574 | tok/s=29,367 | MFU=7.0% | step=142825ms | tokens=1.144B | mem=4.2/32.7GB | cost=$26.72/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 06:51 (131.1h) | Phase3: SKIPPED | SFT: Oct 27 11:35 (135.8h) | DPO (Final): Oct 27 12:09 (136.4h) [2026-10-21 19:47:45][INFO][training.pretrain_engine] Starting global step 327 (accumulating 32 micro-batches)... [2026-10-21 19:47:45][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 19:48:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8590 [2026-10-21 19:48:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4876 [2026-10-21 19:48:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0636 [2026-10-21 19:48:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7668 [2026-10-21 19:48:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8074 [2026-10-21 19:48:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3372 [2026-10-21 19:48:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9704 [2026-10-21 19:48:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1313 [2026-10-21 19:48:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6055 [2026-10-21 19:48:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9792 [2026-10-21 19:48:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9301 [2026-10-21 19:48:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4838 [2026-10-21 19:48:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4196 [2026-10-21 19:48:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6446 [2026-10-21 19:48:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8346 [2026-10-21 19:48:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9879 [2026-10-21 19:49:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1723 [2026-10-21 19:49:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2325 [2026-10-21 19:49:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8076 [2026-10-21 19:49:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5866 [2026-10-21 19:49:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1711 [2026-10-21 19:49:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4051 [2026-10-21 19:49:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6378 [2026-10-21 19:49:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5344 [2026-10-21 19:49:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9208 [2026-10-21 19:49:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8459 [2026-10-21 19:49:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5239 [2026-10-21 19:49:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7964 [2026-10-21 19:50:02][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,612 | MFU: 1.8% [2026-10-21 19:50:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3438 [2026-10-21 19:50:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0997 [2026-10-21 19:50:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0703 [2026-10-21 19:50:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1421 [2026-10-21 19:50:08][INFO][training.pretrain_engine] [Ph2|Step 327] loss=5.0434 | ppl=154.99 | aux=0.02508 | lr=2.99e-04 | gnorm=0.547 | tok/s=29,366 | MFU=7.0% | step=142826ms | tokens=1.145B | mem=4.2/32.7GB | cost=$26.80/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 06:53 (131.1h) | Phase3: SKIPPED | SFT: Oct 27 11:37 (135.8h) | DPO (Final): Oct 27 12:11 (136.4h) [2026-10-21 19:50:08][INFO][training.pretrain_engine] Starting global step 328 (accumulating 32 micro-batches)... [2026-10-21 19:50:08][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 19:50:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7541 [2026-10-21 19:50:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.8565 [2026-10-21 19:50:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3534 [2026-10-21 19:50:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.1758 [2026-10-21 19:50:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7186 [2026-10-21 19:50:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7537 [2026-10-21 19:50:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0829 [2026-10-21 19:50:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9036 [2026-10-21 19:51:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1613 [2026-10-21 19:51:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1619 [2026-10-21 19:51:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2094 [2026-10-21 19:51:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0477 [2026-10-21 19:51:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5826 [2026-10-21 19:51:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.1440 [2026-10-21 19:51:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7568 [2026-10-21 19:51:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0024 [2026-10-21 19:51:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2143 [2026-10-21 19:51:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5379 [2026-10-21 19:51:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0029 [2026-10-21 19:51:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9345 [2026-10-21 19:51:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4406 [2026-10-21 19:51:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1741 [2026-10-21 19:51:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0648 [2026-10-21 19:51:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1787 [2026-10-21 19:52:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0007 [2026-10-21 19:52:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7899 [2026-10-21 19:52:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8167 [2026-10-21 19:52:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6331 [2026-10-21 19:52:25][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,616 | MFU: 1.8% [2026-10-21 19:52:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1982 [2026-10-21 19:52:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4260 [2026-10-21 19:52:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0570 [2026-10-21 19:52:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3557 [2026-10-21 19:52:31][INFO][training.pretrain_engine] [Ph2|Step 328] loss=5.0678 | ppl=158.82 | aux=0.02507 | lr=2.99e-04 | gnorm=0.504 | tok/s=29,366 | MFU=7.0% | step=142830ms | tokens=1.146B | mem=4.2/32.7GB | cost=$26.88/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 06:55 (131.0h) | Phase3: SKIPPED | SFT: Oct 27 11:39 (135.8h) | DPO (Final): Oct 27 12:13 (136.3h) [2026-10-21 19:52:31][INFO][training.pretrain_engine] Starting global step 329 (accumulating 32 micro-batches)... [2026-10-21 19:52:31][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 19:52:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9757 [2026-10-21 19:52:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8244 [2026-10-21 19:52:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7959 [2026-10-21 19:52:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4222 [2026-10-21 19:53:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9638 [2026-10-21 19:53:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2409 [2026-10-21 19:53:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1953 [2026-10-21 19:53:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3996 [2026-10-21 19:53:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6583 [2026-10-21 19:53:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5386 [2026-10-21 19:53:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0820 [2026-10-21 19:53:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8926 [2026-10-21 19:53:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8188 [2026-10-21 19:53:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9751 [2026-10-21 19:53:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4263 [2026-10-21 19:53:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2061 [2026-10-21 19:54:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1057 [2026-10-21 19:54:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0485 [2026-10-21 19:54:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2140 [2026-10-21 19:54:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6454 [2026-10-21 19:54:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6280 [2026-10-21 19:54:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8128 [2026-10-21 19:54:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9082 [2026-10-21 19:54:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0003 [2026-10-21 19:54:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7222 [2026-10-21 19:54:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8869 [2026-10-21 19:54:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1551 [2026-10-21 19:54:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2542 [2026-10-21 19:54:49][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,599 | MFU: 1.8% [2026-10-21 19:54:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1742 [2026-10-21 19:54:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4805 [2026-10-21 19:54:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4297 [2026-10-21 19:54:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1288 [2026-10-21 19:54:55][INFO][training.pretrain_engine] [Ph2|Step 329] loss=5.0688 | ppl=158.99 | aux=0.02506 | lr=2.99e-04 | gnorm=0.504 | tok/s=29,364 | MFU=7.0% | step=142838ms | tokens=1.147B | mem=4.2/32.7GB | cost=$26.97/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 06:57 (131.0h) | Phase3: SKIPPED | SFT: Oct 27 11:41 (135.8h) | DPO (Final): Oct 27 12:15 (136.3h) [2026-10-21 19:54:55][INFO][training.pretrain_engine] Starting global step 330 (accumulating 32 micro-batches)... [2026-10-21 19:54:55][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 19:55:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3114 [2026-10-21 19:55:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1430 [2026-10-21 19:55:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9023 [2026-10-21 19:55:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9892 [2026-10-21 19:55:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7392 [2026-10-21 19:55:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1151 [2026-10-21 19:55:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1736 [2026-10-21 19:55:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3164 [2026-10-21 19:55:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4276 [2026-10-21 19:55:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6081 [2026-10-21 19:55:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2929 [2026-10-21 19:55:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5957 [2026-10-21 19:56:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2177 [2026-10-21 19:56:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3213 [2026-10-21 19:56:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1403 [2026-10-21 19:56:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9380 [2026-10-21 19:56:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1298 [2026-10-21 19:56:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1509 [2026-10-21 19:56:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0830 [2026-10-21 19:56:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5088 [2026-10-21 19:56:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1606 [2026-10-21 19:56:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7771 [2026-10-21 19:56:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9086 [2026-10-21 19:56:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0304 [2026-10-21 19:56:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1218 [2026-10-21 19:56:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4349 [2026-10-21 19:57:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7798 [2026-10-21 19:57:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6065 [2026-10-21 19:57:13][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,605 | MFU: 1.8% [2026-10-21 19:57:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7322 [2026-10-21 19:57:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6424 [2026-10-21 19:57:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2265 [2026-10-21 19:57:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4703 [2026-10-21 19:57:18][INFO][training.pretrain_engine] [Ph2|Step 330] loss=5.0922 | ppl=162.74 | aux=0.02507 | lr=2.99e-04 | gnorm=0.473 | tok/s=29,363 | MFU=7.0% | step=142842ms | tokens=1.148B | mem=4.2/32.7GB | cost=$27.05/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 06:59 (131.0h) | Phase3: SKIPPED | SFT: Oct 27 11:43 (135.8h) | DPO (Final): Oct 27 12:17 (136.3h) [2026-10-21 19:57:18][INFO][training.pretrain_engine] Starting global step 331 (accumulating 32 micro-batches)... [2026-10-21 19:57:18][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 19:57:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6016 [2026-10-21 19:57:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9442 [2026-10-21 19:57:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5168 [2026-10-21 19:57:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2753 [2026-10-21 19:57:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5050 [2026-10-21 19:57:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6903 [2026-10-21 19:57:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.0834 [2026-10-21 19:57:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9762 [2026-10-21 19:58:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9760 [2026-10-21 19:58:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8783 [2026-10-21 19:58:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0058 [2026-10-21 19:58:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.1011 [2026-10-21 19:58:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0053 [2026-10-21 19:58:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3301 [2026-10-21 19:58:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0205 [2026-10-21 19:58:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0705 [2026-10-21 19:58:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2198 [2026-10-21 19:58:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0700 [2026-10-21 19:58:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0420 [2026-10-21 19:58:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0793 [2026-10-21 19:59:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1866 [2026-10-21 19:59:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3419 [2026-10-21 19:59:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9208 [2026-10-21 19:59:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0485 [2026-10-21 19:59:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4375 [2026-10-21 19:59:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.1180 [2026-10-21 19:59:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9406 [2026-10-21 19:59:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=3.6554 [2026-10-21 19:59:36][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,591 | MFU: 1.8% [2026-10-21 19:59:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1131 [2026-10-21 19:59:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6224 [2026-10-21 19:59:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2696 [2026-10-21 19:59:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4886 [2026-10-21 19:59:41][INFO][training.pretrain_engine] [Ph2|Step 331] loss=5.0825 | ppl=161.17 | aux=0.02507 | lr=2.99e-04 | gnorm=0.455 | tok/s=29,362 | MFU=7.0% | step=142849ms | tokens=1.149B | mem=4.2/32.7GB | cost=$27.13/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 07:01 (131.0h) | Phase3: SKIPPED | SFT: Oct 27 11:45 (135.8h) | DPO (Final): Oct 27 12:19 (136.3h) [2026-10-21 19:59:41][INFO][training.pretrain_engine] Starting global step 332 (accumulating 32 micro-batches)... [2026-10-21 19:59:41][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 19:59:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0737 [2026-10-21 19:59:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5710 [2026-10-21 19:59:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5502 [2026-10-21 19:59:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4363 [2026-10-21 20:00:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9560 [2026-10-21 20:00:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8993 [2026-10-21 20:00:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9653 [2026-10-21 20:00:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9973 [2026-10-21 20:00:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5928 [2026-10-21 20:00:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6502 [2026-10-21 20:00:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6369 [2026-10-21 20:00:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1252 [2026-10-21 20:00:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5823 [2026-10-21 20:00:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3565 [2026-10-21 20:00:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7887 [2026-10-21 20:00:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0163 [2026-10-21 20:01:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6529 [2026-10-21 20:01:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8603 [2026-10-21 20:01:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5806 [2026-10-21 20:01:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8551 [2026-10-21 20:01:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=3.9174 [2026-10-21 20:01:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7486 [2026-10-21 20:01:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8948 [2026-10-21 20:01:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9426 [2026-10-21 20:01:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0401 [2026-10-21 20:01:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1326 [2026-10-21 20:01:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3113 [2026-10-21 20:01:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0929 [2026-10-21 20:01:59][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,590 | MFU: 1.8% [2026-10-21 20:02:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5622 [2026-10-21 20:02:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1216 [2026-10-21 20:02:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8553 [2026-10-21 20:02:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4492 [2026-10-21 20:02:05][INFO][training.pretrain_engine] [Ph2|Step 332] loss=5.0717 | ppl=159.44 | aux=0.02507 | lr=2.99e-04 | gnorm=0.496 | tok/s=29,361 | MFU=7.0% | step=142854ms | tokens=1.150B | mem=4.2/32.7GB | cost=$27.21/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 07:03 (131.0h) | Phase3: SKIPPED | SFT: Oct 27 11:47 (135.8h) | DPO (Final): Oct 27 12:21 (136.3h) [2026-10-21 20:02:05][INFO][training.pretrain_engine] Starting global step 333 (accumulating 32 micro-batches)... [2026-10-21 20:02:05][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 20:02:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2262 [2026-10-21 20:02:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2604 [2026-10-21 20:02:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2692 [2026-10-21 20:02:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3737 [2026-10-21 20:02:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6361 [2026-10-21 20:02:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7215 [2026-10-21 20:02:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3190 [2026-10-21 20:02:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6277 [2026-10-21 20:02:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0094 [2026-10-21 20:02:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8928 [2026-10-21 20:02:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7404 [2026-10-21 20:02:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5114 [2026-10-21 20:03:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1166 [2026-10-21 20:03:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7447 [2026-10-21 20:03:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1433 [2026-10-21 20:03:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5954 [2026-10-21 20:03:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1675 [2026-10-21 20:03:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2777 [2026-10-21 20:03:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1086 [2026-10-21 20:03:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4108 [2026-10-21 20:03:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9675 [2026-10-21 20:03:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5771 [2026-10-21 20:03:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8847 [2026-10-21 20:03:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4075 [2026-10-21 20:04:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1766 [2026-10-21 20:04:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0464 [2026-10-21 20:04:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9681 [2026-10-21 20:04:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8923 [2026-10-21 20:04:22][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,602 | MFU: 1.8% [2026-10-21 20:04:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6014 [2026-10-21 20:04:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4720 [2026-10-21 20:04:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3485 [2026-10-21 20:04:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1661 [2026-10-21 20:04:28][INFO][training.pretrain_engine] [Ph2|Step 333] loss=5.0849 | ppl=161.56 | aux=0.02506 | lr=2.99e-04 | gnorm=0.447 | tok/s=29,358 | MFU=7.0% | step=142865ms | tokens=1.151B | mem=4.2/32.7GB | cost=$27.29/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 07:06 (131.0h) | Phase3: SKIPPED | SFT: Oct 27 11:50 (135.8h) | DPO (Final): Oct 27 12:24 (136.3h) [2026-10-21 20:04:32][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step333 | Tokens: 1.151B | Spent: $27.30 | Remaining: $972.70 [2026-10-21 20:04:32][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step333 [2026-10-21 20:04:32][INFO][training.pretrain_engine] Starting global step 334 (accumulating 32 micro-batches)... [2026-10-21 20:04:32][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 20:04:32][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 20:04:32][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 20:04:32][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 20:04:32][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step333 to Phase-Technologies/forge-500m-pretrain (path: phase2_step333)... [2026-10-21 20:04:32][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 20:04:41][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 20:04:41][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step333 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 20:04:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5658 [2026-10-21 20:04:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6247 [2026-10-21 20:04:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5036 [2026-10-21 20:04:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1386 [2026-10-21 20:05:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5657 [2026-10-21 20:05:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6439 [2026-10-21 20:05:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1399 [2026-10-21 20:05:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2423 [2026-10-21 20:05:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7721 [2026-10-21 20:05:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1408 [2026-10-21 20:05:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3263 [2026-10-21 20:05:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9348 [2026-10-21 20:05:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2748 [2026-10-21 20:05:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0444 [2026-10-21 20:05:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1193 [2026-10-21 20:05:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8523 [2026-10-21 20:05:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9786 [2026-10-21 20:05:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9408 [2026-10-21 20:05:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0094 [2026-10-21 20:06:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3081 [2026-10-21 20:06:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8601 [2026-10-21 20:06:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7031 [2026-10-21 20:06:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0669 [2026-10-21 20:06:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3049 [2026-10-21 20:06:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3011 [2026-10-21 20:06:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5544 [2026-10-21 20:06:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5323 [2026-10-21 20:06:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2742 [2026-10-21 20:06:50][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,580 | MFU: 1.8% [2026-10-21 20:06:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6569 [2026-10-21 20:06:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9973 [2026-10-21 20:06:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8942 [2026-10-21 20:06:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5921 [2026-10-21 20:06:54][INFO][training.pretrain_engine] [Ph2|Step 334] loss=5.0325 | ppl=153.32 | aux=0.02506 | lr=2.99e-04 | gnorm=0.428 | tok/s=29,358 | MFU=7.0% | step=142869ms | tokens=1.152B | mem=4.2/32.7GB | cost=$27.38/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 07:08 (131.0h) | Phase3: SKIPPED | SFT: Oct 27 11:52 (135.8h) | DPO (Final): Oct 27 12:26 (136.3h) [2026-10-21 20:06:54][INFO][training.pretrain_engine] Starting global step 335 (accumulating 32 micro-batches)... [2026-10-21 20:06:54][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 20:07:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6653 [2026-10-21 20:07:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2944 [2026-10-21 20:07:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8940 [2026-10-21 20:07:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9902 [2026-10-21 20:07:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9360 [2026-10-21 20:07:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9750 [2026-10-21 20:07:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1372 [2026-10-21 20:07:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3809 [2026-10-21 20:07:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3687 [2026-10-21 20:07:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7954 [2026-10-21 20:07:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1671 [2026-10-21 20:07:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6889 [2026-10-21 20:08:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4389 [2026-10-21 20:08:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0509 [2026-10-21 20:08:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1520 [2026-10-21 20:08:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8421 [2026-10-21 20:08:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1009 [2026-10-21 20:08:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1593 [2026-10-21 20:08:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3323 [2026-10-21 20:08:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3827 [2026-10-21 20:08:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9482 [2026-10-21 20:08:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7324 [2026-10-21 20:08:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6677 [2026-10-21 20:08:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3960 [2026-10-21 20:08:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5160 [2026-10-21 20:08:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4309 [2026-10-21 20:09:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2948 [2026-10-21 20:09:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7846 [2026-10-21 20:09:12][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,595 | MFU: 1.8% [2026-10-21 20:09:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1387 [2026-10-21 20:09:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2556 [2026-10-21 20:09:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8893 [2026-10-21 20:09:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4621 [2026-10-21 20:09:18][INFO][training.pretrain_engine] [Ph2|Step 335] loss=5.0370 | ppl=154.00 | aux=0.02507 | lr=2.99e-04 | gnorm=0.449 | tok/s=29,355 | MFU=7.0% | step=142880ms | tokens=1.153B | mem=4.2/32.7GB | cost=$27.46/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 07:10 (131.0h) | Phase3: SKIPPED | SFT: Oct 27 11:54 (135.8h) | DPO (Final): Oct 27 12:28 (136.3h) [2026-10-21 20:09:18][INFO][training.pretrain_engine] Starting global step 336 (accumulating 32 micro-batches)... [2026-10-21 20:09:18][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 20:09:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1793 [2026-10-21 20:09:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3007 [2026-10-21 20:09:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1229 [2026-10-21 20:09:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2712 [2026-10-21 20:09:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0641 [2026-10-21 20:09:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2207 [2026-10-21 20:09:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3053 [2026-10-21 20:09:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5163 [2026-10-21 20:10:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0000 [2026-10-21 20:10:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9549 [2026-10-21 20:10:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8073 [2026-10-21 20:10:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2183 [2026-10-21 20:10:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1331 [2026-10-21 20:10:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4181 [2026-10-21 20:10:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2349 [2026-10-21 20:10:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0546 [2026-10-21 20:10:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9002 [2026-10-21 20:10:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3080 [2026-10-21 20:10:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4318 [2026-10-21 20:10:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9083 [2026-10-21 20:11:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9891 [2026-10-21 20:11:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3797 [2026-10-21 20:11:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4547 [2026-10-21 20:11:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1567 [2026-10-21 20:11:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4709 [2026-10-21 20:11:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3125 [2026-10-21 20:11:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4247 [2026-10-21 20:11:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0858 [2026-10-21 20:11:36][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,615 | MFU: 1.8% [2026-10-21 20:11:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9944 [2026-10-21 20:11:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0990 [2026-10-21 20:11:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1906 [2026-10-21 20:11:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1242 [2026-10-21 20:11:41][INFO][training.pretrain_engine] [Ph2|Step 336] loss=5.0793 | ppl=160.66 | aux=0.02506 | lr=2.99e-04 | gnorm=0.482 | tok/s=29,355 | MFU=7.0% | step=142881ms | tokens=1.154B | mem=4.2/32.7GB | cost=$27.54/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 07:12 (131.0h) | Phase3: SKIPPED | SFT: Oct 27 11:56 (135.7h) | DPO (Final): Oct 27 12:30 (136.3h) [2026-10-21 20:11:41][INFO][training.pretrain_engine] Starting global step 337 (accumulating 32 micro-batches)... [2026-10-21 20:11:41][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 20:11:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0009 [2026-10-21 20:11:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7828 [2026-10-21 20:11:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9255 [2026-10-21 20:11:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9909 [2026-10-21 20:12:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6335 [2026-10-21 20:12:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3384 [2026-10-21 20:12:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1972 [2026-10-21 20:12:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2377 [2026-10-21 20:12:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0686 [2026-10-21 20:12:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2458 [2026-10-21 20:12:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2523 [2026-10-21 20:12:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7298 [2026-10-21 20:12:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9536 [2026-10-21 20:12:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5135 [2026-10-21 20:12:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9342 [2026-10-21 20:12:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1094 [2026-10-21 20:13:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4209 [2026-10-21 20:13:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3899 [2026-10-21 20:13:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4124 [2026-10-21 20:13:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6378 [2026-10-21 20:13:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1624 [2026-10-21 20:13:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0489 [2026-10-21 20:13:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4803 [2026-10-21 20:13:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3831 [2026-10-21 20:13:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0508 [2026-10-21 20:13:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4834 [2026-10-21 20:13:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0220 [2026-10-21 20:13:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0727 [2026-10-21 20:13:59][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,613 | MFU: 1.8% [2026-10-21 20:14:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9516 [2026-10-21 20:14:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1480 [2026-10-21 20:14:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4412 [2026-10-21 20:14:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3396 [2026-10-21 20:14:04][INFO][training.pretrain_engine] [Ph2|Step 337] loss=5.1507 | ppl=172.55 | aux=0.02506 | lr=2.99e-04 | gnorm=0.451 | tok/s=29,354 | MFU=7.0% | step=142886ms | tokens=1.156B | mem=4.2/32.7GB | cost=$27.62/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 07:14 (131.0h) | Phase3: SKIPPED | SFT: Oct 27 11:58 (135.7h) | DPO (Final): Oct 27 12:32 (136.3h) [2026-10-21 20:14:04][INFO][training.pretrain_engine] Starting global step 338 (accumulating 32 micro-batches)... [2026-10-21 20:14:04][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 20:14:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0785 [2026-10-21 20:14:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5200 [2026-10-21 20:14:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9920 [2026-10-21 20:14:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5013 [2026-10-21 20:14:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7698 [2026-10-21 20:14:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0177 [2026-10-21 20:14:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7004 [2026-10-21 20:14:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7626 [2026-10-21 20:14:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7938 [2026-10-21 20:14:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6134 [2026-10-21 20:14:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3748 [2026-10-21 20:14:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7841 [2026-10-21 20:15:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7510 [2026-10-21 20:15:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9356 [2026-10-21 20:15:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6447 [2026-10-21 20:15:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5512 [2026-10-21 20:15:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5067 [2026-10-21 20:15:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7286 [2026-10-21 20:15:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9351 [2026-10-21 20:15:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2130 [2026-10-21 20:15:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2001 [2026-10-21 20:15:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6705 [2026-10-21 20:15:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8587 [2026-10-21 20:15:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0207 [2026-10-21 20:16:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7019 [2026-10-21 20:16:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1902 [2026-10-21 20:16:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3387 [2026-10-21 20:16:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2436 [2026-10-21 20:16:22][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,621 | MFU: 1.8% [2026-10-21 20:16:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5865 [2026-10-21 20:16:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7653 [2026-10-21 20:16:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8985 [2026-10-21 20:16:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9110 [2026-10-21 20:16:27][INFO][training.pretrain_engine] [Ph2|Step 338] loss=5.2167 | ppl=184.33 | aux=0.02507 | lr=2.99e-04 | gnorm=0.414 | tok/s=29,353 | MFU=7.0% | step=142890ms | tokens=1.157B | mem=4.2/32.7GB | cost=$27.71/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 07:16 (131.0h) | Phase3: SKIPPED | SFT: Oct 27 12:00 (135.7h) | DPO (Final): Oct 27 12:34 (136.3h) [2026-10-21 20:16:27][INFO][training.pretrain_engine] Starting global step 339 (accumulating 32 micro-batches)... [2026-10-21 20:16:27][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 20:16:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0469 [2026-10-21 20:16:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6991 [2026-10-21 20:16:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5370 [2026-10-21 20:16:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3737 [2026-10-21 20:17:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9039 [2026-10-21 20:17:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3046 [2026-10-21 20:17:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.6598 [2026-10-21 20:17:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0611 [2026-10-21 20:17:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6976 [2026-10-21 20:17:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0378 [2026-10-21 20:17:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2691 [2026-10-21 20:17:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0660 [2026-10-21 20:17:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1361 [2026-10-21 20:17:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5832 [2026-10-21 20:17:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9744 [2026-10-21 20:17:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8270 [2026-10-21 20:17:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4551 [2026-10-21 20:17:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0174 [2026-10-21 20:17:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3524 [2026-10-21 20:17:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1974 [2026-10-21 20:18:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6090 [2026-10-21 20:18:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7105 [2026-10-21 20:18:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6993 [2026-10-21 20:18:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5173 [2026-10-21 20:18:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4853 [2026-10-21 20:18:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6510 [2026-10-21 20:18:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6651 [2026-10-21 20:18:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3769 [2026-10-21 20:18:45][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,622 | MFU: 1.8% [2026-10-21 20:18:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4541 [2026-10-21 20:18:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8185 [2026-10-21 20:18:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6303 [2026-10-21 20:18:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0245 [2026-10-21 20:18:50][INFO][training.pretrain_engine] [Ph2|Step 339] loss=5.2160 | ppl=184.19 | aux=0.02507 | lr=2.99e-04 | gnorm=0.494 | tok/s=29,353 | MFU=7.0% | step=142890ms | tokens=1.158B | mem=4.2/32.7GB | cost=$27.79/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 07:18 (131.0h) | Phase3: SKIPPED | SFT: Oct 27 12:02 (135.7h) | DPO (Final): Oct 27 12:36 (136.3h) [2026-10-21 20:18:50][INFO][training.pretrain_engine] Starting global step 340 (accumulating 32 micro-batches)... [2026-10-21 20:18:50][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 20:19:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2479 [2026-10-21 20:19:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1162 [2026-10-21 20:19:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3377 [2026-10-21 20:19:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9517 [2026-10-21 20:19:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2632 [2026-10-21 20:19:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3873 [2026-10-21 20:19:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1825 [2026-10-21 20:19:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0660 [2026-10-21 20:19:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8361 [2026-10-21 20:19:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4281 [2026-10-21 20:19:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3075 [2026-10-21 20:19:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2244 [2026-10-21 20:20:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1669 [2026-10-21 20:20:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7187 [2026-10-21 20:20:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2335 [2026-10-21 20:20:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6475 [2026-10-21 20:20:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1864 [2026-10-21 20:20:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0204 [2026-10-21 20:20:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9430 [2026-10-21 20:20:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4138 [2026-10-21 20:20:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2536 [2026-10-21 20:20:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8434 [2026-10-21 20:20:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4385 [2026-10-21 20:20:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4604 [2026-10-21 20:20:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5694 [2026-10-21 20:20:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5024 [2026-10-21 20:20:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7781 [2026-10-21 20:20:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3707 [2026-10-21 20:21:09][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,590 | MFU: 1.8% [2026-10-21 20:21:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9881 [2026-10-21 20:21:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5466 [2026-10-21 20:21:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4378 [2026-10-21 20:21:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1603 [2026-10-21 20:21:14][INFO][training.pretrain_engine] [Ph2|Step 340] loss=5.0790 | ppl=160.62 | aux=0.02507 | lr=2.99e-04 | gnorm=0.648 | tok/s=29,352 | MFU=7.0% | step=142897ms | tokens=1.159B | mem=4.2/32.7GB | cost=$27.87/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 07:20 (131.0h) | Phase3: SKIPPED | SFT: Oct 27 12:04 (135.7h) | DPO (Final): Oct 27 12:38 (136.3h) [2026-10-21 20:21:14][INFO][training.pretrain_engine] Starting global step 341 (accumulating 32 micro-batches)... [2026-10-21 20:21:14][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 20:21:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4597 [2026-10-21 20:21:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3834 [2026-10-21 20:21:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2495 [2026-10-21 20:21:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8205 [2026-10-21 20:21:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5792 [2026-10-21 20:21:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3163 [2026-10-21 20:21:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2696 [2026-10-21 20:21:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9696 [2026-10-21 20:22:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8498 [2026-10-21 20:22:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0725 [2026-10-21 20:22:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1433 [2026-10-21 20:22:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3663 [2026-10-21 20:22:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4210 [2026-10-21 20:22:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6125 [2026-10-21 20:22:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5020 [2026-10-21 20:22:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6403 [2026-10-21 20:22:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7965 [2026-10-21 20:22:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0766 [2026-10-21 20:22:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9917 [2026-10-21 20:22:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7393 [2026-10-21 20:23:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9752 [2026-10-21 20:23:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0886 [2026-10-21 20:23:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3269 [2026-10-21 20:23:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3173 [2026-10-21 20:23:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3982 [2026-10-21 20:23:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3775 [2026-10-21 20:23:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6670 [2026-10-21 20:23:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9276 [2026-10-21 20:23:32][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,607 | MFU: 1.8% [2026-10-21 20:23:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4874 [2026-10-21 20:23:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3402 [2026-10-21 20:23:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6731 [2026-10-21 20:23:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8847 [2026-10-21 20:23:37][INFO][training.pretrain_engine] [Ph2|Step 341] loss=5.0445 | ppl=155.17 | aux=0.02506 | lr=2.99e-04 | gnorm=0.820 | tok/s=29,350 | MFU=7.0% | step=142905ms | tokens=1.160B | mem=4.2/32.7GB | cost=$27.95/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 07:22 (131.0h) | Phase3: SKIPPED | SFT: Oct 27 12:06 (135.7h) | DPO (Final): Oct 27 12:40 (136.3h) [2026-10-21 20:23:37][INFO][training.pretrain_engine] Starting global step 342 (accumulating 32 micro-batches)... [2026-10-21 20:23:37][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 20:23:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7181 [2026-10-21 20:23:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7040 [2026-10-21 20:23:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3203 [2026-10-21 20:23:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9231 [2026-10-21 20:24:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3114 [2026-10-21 20:24:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0574 [2026-10-21 20:24:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8048 [2026-10-21 20:24:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3733 [2026-10-21 20:24:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9085 [2026-10-21 20:24:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2805 [2026-10-21 20:24:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3543 [2026-10-21 20:24:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1884 [2026-10-21 20:24:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6607 [2026-10-21 20:24:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5700 [2026-10-21 20:24:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8376 [2026-10-21 20:24:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7906 [2026-10-21 20:25:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9445 [2026-10-21 20:25:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8487 [2026-10-21 20:25:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6215 [2026-10-21 20:25:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6611 [2026-10-21 20:25:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4867 [2026-10-21 20:25:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8004 [2026-10-21 20:25:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6672 [2026-10-21 20:25:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4483 [2026-10-21 20:25:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6165 [2026-10-21 20:25:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3268 [2026-10-21 20:25:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4214 [2026-10-21 20:25:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4307 [2026-10-21 20:25:56][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,581 | MFU: 1.8% [2026-10-21 20:26:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1689 [2026-10-21 20:26:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0744 [2026-10-21 20:26:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7925 [2026-10-21 20:26:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5023 [2026-10-21 20:26:00][INFO][training.pretrain_engine] [Ph2|Step 342] loss=5.0280 | ppl=152.63 | aux=0.02506 | lr=2.99e-04 | gnorm=0.957 | tok/s=29,350 | MFU=7.0% | step=142906ms | tokens=1.161B | mem=4.2/32.7GB | cost=$28.03/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 07:24 (131.0h) | Phase3: SKIPPED | SFT: Oct 27 12:08 (135.7h) | DPO (Final): Oct 27 12:42 (136.3h) [2026-10-21 20:26:04][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step342 | Tokens: 1.161B | Spent: $28.03 | Remaining: $971.97 [2026-10-21 20:26:04][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step342 [2026-10-21 20:26:04][INFO][training.pretrain_engine] Starting global step 343 (accumulating 32 micro-batches)... [2026-10-21 20:26:04][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 20:26:04][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 20:26:04][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 20:26:04][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 20:26:04][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step342 to Phase-Technologies/forge-500m-pretrain (path: phase2_step342)... [2026-10-21 20:26:04][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 20:26:13][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 20:26:13][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step342 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 20:26:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9130 [2026-10-21 20:26:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3145 [2026-10-21 20:26:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0276 [2026-10-21 20:26:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4227 [2026-10-21 20:26:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6088 [2026-10-21 20:26:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3630 [2026-10-21 20:26:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1312 [2026-10-21 20:26:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8914 [2026-10-21 20:26:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1673 [2026-10-21 20:26:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0224 [2026-10-21 20:26:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3560 [2026-10-21 20:26:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2457 [2026-10-21 20:27:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6500 [2026-10-21 20:27:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6109 [2026-10-21 20:27:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1559 [2026-10-21 20:27:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2892 [2026-10-21 20:27:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9285 [2026-10-21 20:27:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4280 [2026-10-21 20:27:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2986 [2026-10-21 20:27:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0192 [2026-10-21 20:27:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0568 [2026-10-21 20:27:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1358 [2026-10-21 20:27:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0561 [2026-10-21 20:27:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3388 [2026-10-21 20:28:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1831 [2026-10-21 20:28:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8695 [2026-10-21 20:28:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8710 [2026-10-21 20:28:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8154 [2026-10-21 20:28:22][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,607 | MFU: 1.8% [2026-10-21 20:28:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7878 [2026-10-21 20:28:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7838 [2026-10-21 20:28:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9690 [2026-10-21 20:28:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1173 [2026-10-21 20:28:26][INFO][training.pretrain_engine] [Ph2|Step 343] loss=5.0156 | ppl=150.74 | aux=0.02506 | lr=2.99e-04 | gnorm=0.891 | tok/s=29,352 | MFU=7.0% | step=142899ms | tokens=1.162B | mem=4.2/32.7GB | cost=$28.12/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 07:26 (131.0h) | Phase3: SKIPPED | SFT: Oct 27 12:10 (135.7h) | DPO (Final): Oct 27 12:44 (136.3h) [2026-10-21 20:28:26][INFO][training.pretrain_engine] Starting global step 344 (accumulating 32 micro-batches)... [2026-10-21 20:28:26][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 20:28:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3849 [2026-10-21 20:28:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0918 [2026-10-21 20:28:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1210 [2026-10-21 20:28:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7822 [2026-10-21 20:29:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3082 [2026-10-21 20:29:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1076 [2026-10-21 20:29:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2665 [2026-10-21 20:29:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8588 [2026-10-21 20:29:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2278 [2026-10-21 20:29:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1389 [2026-10-21 20:29:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9748 [2026-10-21 20:29:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2998 [2026-10-21 20:29:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9244 [2026-10-21 20:29:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7460 [2026-10-21 20:29:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2482 [2026-10-21 20:29:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6169 [2026-10-21 20:29:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7606 [2026-10-21 20:29:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8666 [2026-10-21 20:29:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8850 [2026-10-21 20:29:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1624 [2026-10-21 20:30:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3593 [2026-10-21 20:30:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3392 [2026-10-21 20:30:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6927 [2026-10-21 20:30:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8297 [2026-10-21 20:30:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9925 [2026-10-21 20:30:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6899 [2026-10-21 20:30:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5944 [2026-10-21 20:30:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2328 [2026-10-21 20:30:44][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,600 | MFU: 1.8% [2026-10-21 20:30:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8576 [2026-10-21 20:30:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6381 [2026-10-21 20:30:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6290 [2026-10-21 20:30:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1402 [2026-10-21 20:30:49][INFO][training.pretrain_engine] [Ph2|Step 344] loss=5.1318 | ppl=169.32 | aux=0.02507 | lr=2.99e-04 | gnorm=0.926 | tok/s=29,349 | MFU=7.0% | step=142911ms | tokens=1.163B | mem=4.2/32.7GB | cost=$28.20/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 07:28 (131.0h) | Phase3: SKIPPED | SFT: Oct 27 12:12 (135.7h) | DPO (Final): Oct 27 12:46 (136.3h) [2026-10-21 20:30:49][INFO][training.pretrain_engine] Starting global step 345 (accumulating 32 micro-batches)... [2026-10-21 20:30:49][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 20:31:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2525 [2026-10-21 20:31:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4311 [2026-10-21 20:31:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1419 [2026-10-21 20:31:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1183 [2026-10-21 20:31:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0287 [2026-10-21 20:31:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7470 [2026-10-21 20:31:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9566 [2026-10-21 20:31:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8588 [2026-10-21 20:31:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8527 [2026-10-21 20:31:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1725 [2026-10-21 20:31:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0528 [2026-10-21 20:31:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1002 [2026-10-21 20:32:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3419 [2026-10-21 20:32:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4212 [2026-10-21 20:32:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3857 [2026-10-21 20:32:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3576 [2026-10-21 20:32:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3364 [2026-10-21 20:32:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2788 [2026-10-21 20:32:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6031 [2026-10-21 20:32:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4452 [2026-10-21 20:32:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6619 [2026-10-21 20:32:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3556 [2026-10-21 20:32:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4472 [2026-10-21 20:32:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4742 [2026-10-21 20:32:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1387 [2026-10-21 20:32:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7730 [2026-10-21 20:32:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8392 [2026-10-21 20:32:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1828 [2026-10-21 20:33:08][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,593 | MFU: 1.8% [2026-10-21 20:33:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9157 [2026-10-21 20:33:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2040 [2026-10-21 20:33:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4544 [2026-10-21 20:33:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4681 [2026-10-21 20:33:13][INFO][training.pretrain_engine] [Ph2|Step 345] loss=5.1787 | ppl=177.46 | aux=0.02507 | lr=2.99e-04 | gnorm=0.926 | tok/s=29,348 | MFU=7.0% | step=142916ms | tokens=1.164B | mem=4.2/32.7GB | cost=$28.28/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 07:30 (131.0h) | Phase3: SKIPPED | SFT: Oct 27 12:14 (135.7h) | DPO (Final): Oct 27 12:48 (136.3h) [2026-10-21 20:33:13][INFO][training.pretrain_engine] Starting global step 346 (accumulating 32 micro-batches)... [2026-10-21 20:33:13][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 20:33:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0712 [2026-10-21 20:33:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6394 [2026-10-21 20:33:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2422 [2026-10-21 20:33:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1907 [2026-10-21 20:33:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6625 [2026-10-21 20:33:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2501 [2026-10-21 20:33:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0216 [2026-10-21 20:33:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1936 [2026-10-21 20:34:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8664 [2026-10-21 20:34:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6479 [2026-10-21 20:34:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7622 [2026-10-21 20:34:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8835 [2026-10-21 20:34:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9352 [2026-10-21 20:34:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3771 [2026-10-21 20:34:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7583 [2026-10-21 20:34:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5225 [2026-10-21 20:34:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6043 [2026-10-21 20:34:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1151 [2026-10-21 20:34:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3210 [2026-10-21 20:34:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8022 [2026-10-21 20:34:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2215 [2026-10-21 20:35:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1635 [2026-10-21 20:35:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3914 [2026-10-21 20:35:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0622 [2026-10-21 20:35:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7723 [2026-10-21 20:35:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9751 [2026-10-21 20:35:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7442 [2026-10-21 20:35:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6780 [2026-10-21 20:35:31][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,596 | MFU: 1.8% [2026-10-21 20:35:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0780 [2026-10-21 20:35:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0851 [2026-10-21 20:35:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9820 [2026-10-21 20:35:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0674 [2026-10-21 20:35:36][INFO][training.pretrain_engine] [Ph2|Step 346] loss=5.1349 | ppl=169.84 | aux=0.02506 | lr=2.99e-04 | gnorm=0.934 | tok/s=29,349 | MFU=7.0% | step=142910ms | tokens=1.165B | mem=4.2/32.7GB | cost=$28.36/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 07:32 (130.9h) | Phase3: SKIPPED | SFT: Oct 27 12:16 (135.7h) | DPO (Final): Oct 27 12:50 (136.2h) [2026-10-21 20:35:36][INFO][training.pretrain_engine] Starting global step 347 (accumulating 32 micro-batches)... [2026-10-21 20:35:36][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 20:35:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5235 [2026-10-21 20:35:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8983 [2026-10-21 20:35:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1657 [2026-10-21 20:35:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6675 [2026-10-21 20:36:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2699 [2026-10-21 20:36:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9515 [2026-10-21 20:36:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9432 [2026-10-21 20:36:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2379 [2026-10-21 20:36:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5276 [2026-10-21 20:36:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9007 [2026-10-21 20:36:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4799 [2026-10-21 20:36:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1511 [2026-10-21 20:36:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9129 [2026-10-21 20:36:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3690 [2026-10-21 20:36:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6519 [2026-10-21 20:36:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2767 [2026-10-21 20:37:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3414 [2026-10-21 20:37:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8813 [2026-10-21 20:37:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4081 [2026-10-21 20:37:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0904 [2026-10-21 20:37:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8324 [2026-10-21 20:37:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3605 [2026-10-21 20:37:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1683 [2026-10-21 20:37:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0830 [2026-10-21 20:37:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3614 [2026-10-21 20:37:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5015 [2026-10-21 20:37:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8034 [2026-10-21 20:37:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2076 [2026-10-21 20:37:54][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,578 | MFU: 1.8% [2026-10-21 20:37:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9188 [2026-10-21 20:37:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6151 [2026-10-21 20:37:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1733 [2026-10-21 20:37:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8757 [2026-10-21 20:37:59][INFO][training.pretrain_engine] [Ph2|Step 347] loss=5.1005 | ppl=164.10 | aux=0.02506 | lr=2.99e-04 | gnorm=0.949 | tok/s=29,350 | MFU=7.0% | step=142907ms | tokens=1.166B | mem=4.2/32.7GB | cost=$28.44/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 07:33 (130.9h) | Phase3: SKIPPED | SFT: Oct 27 12:17 (135.7h) | DPO (Final): Oct 27 12:51 (136.2h) [2026-10-21 20:37:59][INFO][training.pretrain_engine] Starting global step 348 (accumulating 32 micro-batches)... [2026-10-21 20:37:59][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 20:38:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8143 [2026-10-21 20:38:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0923 [2026-10-21 20:38:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4517 [2026-10-21 20:38:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9086 [2026-10-21 20:38:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1192 [2026-10-21 20:38:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0710 [2026-10-21 20:38:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1118 [2026-10-21 20:38:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1789 [2026-10-21 20:38:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1844 [2026-10-21 20:38:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9145 [2026-10-21 20:38:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7407 [2026-10-21 20:38:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8799 [2026-10-21 20:39:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0148 [2026-10-21 20:39:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1299 [2026-10-21 20:39:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4124 [2026-10-21 20:39:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8220 [2026-10-21 20:39:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4784 [2026-10-21 20:39:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3702 [2026-10-21 20:39:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8553 [2026-10-21 20:39:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0927 [2026-10-21 20:39:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4031 [2026-10-21 20:39:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9597 [2026-10-21 20:39:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0147 [2026-10-21 20:39:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0935 [2026-10-21 20:40:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4740 [2026-10-21 20:40:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8793 [2026-10-21 20:40:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4414 [2026-10-21 20:40:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4702 [2026-10-21 20:40:17][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,597 | MFU: 1.8% [2026-10-21 20:40:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0622 [2026-10-21 20:40:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1174 [2026-10-21 20:40:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9394 [2026-10-21 20:40:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9170 [2026-10-21 20:40:22][INFO][training.pretrain_engine] [Ph2|Step 348] loss=5.1880 | ppl=179.11 | aux=0.02506 | lr=2.99e-04 | gnorm=0.789 | tok/s=29,349 | MFU=7.0% | step=142909ms | tokens=1.167B | mem=4.2/32.7GB | cost=$28.53/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 07:35 (130.9h) | Phase3: SKIPPED | SFT: Oct 27 12:19 (135.7h) | DPO (Final): Oct 27 12:53 (136.2h) [2026-10-21 20:40:22][INFO][training.pretrain_engine] Starting global step 349 (accumulating 32 micro-batches)... [2026-10-21 20:40:22][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 20:40:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9655 [2026-10-21 20:40:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4046 [2026-10-21 20:40:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8004 [2026-10-21 20:40:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8807 [2026-10-21 20:40:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5734 [2026-10-21 20:40:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0073 [2026-10-21 20:40:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8288 [2026-10-21 20:40:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8392 [2026-10-21 20:41:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1271 [2026-10-21 20:41:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8491 [2026-10-21 20:41:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1353 [2026-10-21 20:41:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7132 [2026-10-21 20:41:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9245 [2026-10-21 20:41:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0953 [2026-10-21 20:41:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9702 [2026-10-21 20:41:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3211 [2026-10-21 20:41:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0679 [2026-10-21 20:41:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3805 [2026-10-21 20:41:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3412 [2026-10-21 20:41:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1435 [2026-10-21 20:42:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8871 [2026-10-21 20:42:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9512 [2026-10-21 20:42:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8854 [2026-10-21 20:42:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0378 [2026-10-21 20:42:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9414 [2026-10-21 20:42:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9542 [2026-10-21 20:42:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7406 [2026-10-21 20:42:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9911 [2026-10-21 20:42:40][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,578 | MFU: 1.8% [2026-10-21 20:42:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8756 [2026-10-21 20:42:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3454 [2026-10-21 20:42:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9449 [2026-10-21 20:42:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8446 [2026-10-21 20:42:45][INFO][training.pretrain_engine] [Ph2|Step 349] loss=5.0663 | ppl=158.58 | aux=0.02506 | lr=2.99e-04 | gnorm=0.738 | tok/s=29,350 | MFU=7.0% | step=142906ms | tokens=1.168B | mem=4.2/32.7GB | cost=$28.61/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 07:37 (130.9h) | Phase3: SKIPPED | SFT: Oct 27 12:21 (135.6h) | DPO (Final): Oct 27 12:55 (136.2h) [2026-10-21 20:42:45][INFO][training.pretrain_engine] Starting global step 350 (accumulating 32 micro-batches)... [2026-10-21 20:42:45][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 20:43:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7376 [2026-10-21 20:43:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1738 [2026-10-21 20:43:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9650 [2026-10-21 20:43:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8162 [2026-10-21 20:43:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5112 [2026-10-21 20:43:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8233 [2026-10-21 20:43:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6243 [2026-10-21 20:43:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8221 [2026-10-21 20:43:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2280 [2026-10-21 20:43:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6815 [2026-10-21 20:43:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6644 [2026-10-21 20:43:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.0961 [2026-10-21 20:43:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3588 [2026-10-21 20:43:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0292 [2026-10-21 20:43:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6116 [2026-10-21 20:43:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9129 [2026-10-21 20:44:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4473 [2026-10-21 20:44:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2723 [2026-10-21 20:44:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4668 [2026-10-21 20:44:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0639 [2026-10-21 20:44:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2274 [2026-10-21 20:44:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7934 [2026-10-21 20:44:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2710 [2026-10-21 20:44:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0218 [2026-10-21 20:44:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4947 [2026-10-21 20:44:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7788 [2026-10-21 20:44:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5671 [2026-10-21 20:44:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1128 [2026-10-21 20:45:03][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,606 | MFU: 1.8% [2026-10-21 20:45:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9715 [2026-10-21 20:45:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7842 [2026-10-21 20:45:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3171 [2026-10-21 20:45:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0022 [2026-10-21 20:45:08][INFO][training.pretrain_engine] [Ph2|Step 350] loss=5.1444 | ppl=171.46 | aux=0.02505 | lr=2.99e-04 | gnorm=0.664 | tok/s=29,352 | MFU=7.0% | step=142899ms | tokens=1.169B | mem=4.2/32.7GB | cost=$28.69/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 07:38 (130.9h) | Phase3: SKIPPED | SFT: Oct 27 12:22 (135.6h) | DPO (Final): Oct 27 12:56 (136.2h) [2026-10-21 20:45:08][INFO][training.pretrain_engine] Starting global step 351 (accumulating 32 micro-batches)... [2026-10-21 20:45:08][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 20:45:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1622 [2026-10-21 20:45:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1974 [2026-10-21 20:45:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1946 [2026-10-21 20:45:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7359 [2026-10-21 20:45:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8838 [2026-10-21 20:45:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9347 [2026-10-21 20:45:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8700 [2026-10-21 20:45:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1260 [2026-10-21 20:46:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8744 [2026-10-21 20:46:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0012 [2026-10-21 20:46:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3587 [2026-10-21 20:46:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1310 [2026-10-21 20:46:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0796 [2026-10-21 20:46:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9340 [2026-10-21 20:46:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6509 [2026-10-21 20:46:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2289 [2026-10-21 20:46:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3444 [2026-10-21 20:46:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3852 [2026-10-21 20:46:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8764 [2026-10-21 20:46:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=3.9504 [2026-10-21 20:46:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4112 [2026-10-21 20:46:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4896 [2026-10-21 20:46:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3017 [2026-10-21 20:46:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8022 [2026-10-21 20:47:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2618 [2026-10-21 20:47:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8032 [2026-10-21 20:47:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2481 [2026-10-21 20:47:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5152 [2026-10-21 20:47:26][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,590 | MFU: 1.8% [2026-10-21 20:47:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8135 [2026-10-21 20:47:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2974 [2026-10-21 20:47:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7399 [2026-10-21 20:47:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1233 [2026-10-21 20:47:30][INFO][training.pretrain_engine] [Ph2|Step 351] loss=5.1321 | ppl=169.38 | aux=0.02505 | lr=2.99e-04 | gnorm=0.602 | tok/s=29,352 | MFU=7.0% | step=142896ms | tokens=1.170B | mem=4.2/32.7GB | cost=$28.77/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 07:40 (130.9h) | Phase3: SKIPPED | SFT: Oct 27 12:24 (135.6h) | DPO (Final): Oct 27 12:58 (136.2h) [2026-10-21 20:47:33][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step351 | Tokens: 1.170B | Spent: $28.77 | Remaining: $971.23 [2026-10-21 20:47:33][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step351 [2026-10-21 20:47:33][INFO][training.pretrain_engine] Starting global step 352 (accumulating 32 micro-batches)... [2026-10-21 20:47:33][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 20:47:33][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 20:47:33][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 20:47:33][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 20:47:33][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step351 to Phase-Technologies/forge-500m-pretrain (path: phase2_step351)... [2026-10-21 20:47:34][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 20:47:43][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 20:47:43][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step351 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 20:47:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8394 [2026-10-21 20:47:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9080 [2026-10-21 20:47:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0688 [2026-10-21 20:47:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7864 [2026-10-21 20:48:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1417 [2026-10-21 20:48:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2880 [2026-10-21 20:48:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4494 [2026-10-21 20:48:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1656 [2026-10-21 20:48:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0572 [2026-10-21 20:48:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5410 [2026-10-21 20:48:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0878 [2026-10-21 20:48:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1991 [2026-10-21 20:48:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0756 [2026-10-21 20:48:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4104 [2026-10-21 20:48:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9116 [2026-10-21 20:48:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4084 [2026-10-21 20:48:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0882 [2026-10-21 20:48:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8303 [2026-10-21 20:48:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8385 [2026-10-21 20:49:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0819 [2026-10-21 20:49:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0177 [2026-10-21 20:49:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9830 [2026-10-21 20:49:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0205 [2026-10-21 20:49:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9566 [2026-10-21 20:49:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0295 [2026-10-21 20:49:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0394 [2026-10-21 20:49:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8456 [2026-10-21 20:49:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0102 [2026-10-21 20:49:51][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,588 | MFU: 1.8% [2026-10-21 20:49:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1284 [2026-10-21 20:49:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8698 [2026-10-21 20:49:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2958 [2026-10-21 20:49:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2851 [2026-10-21 20:49:56][INFO][training.pretrain_engine] [Ph2|Step 352] loss=5.1024 | ppl=164.41 | aux=0.02506 | lr=2.99e-04 | gnorm=0.676 | tok/s=29,353 | MFU=7.0% | step=142893ms | tokens=1.171B | mem=4.2/32.7GB | cost=$28.85/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 07:41 (130.9h) | Phase3: SKIPPED | SFT: Oct 27 12:25 (135.6h) | DPO (Final): Oct 27 12:59 (136.2h) [2026-10-21 20:49:56][INFO][training.pretrain_engine] Starting global step 353 (accumulating 32 micro-batches)... [2026-10-21 20:49:56][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 20:50:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5958 [2026-10-21 20:50:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2119 [2026-10-21 20:50:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9703 [2026-10-21 20:50:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2872 [2026-10-21 20:50:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0360 [2026-10-21 20:50:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9221 [2026-10-21 20:50:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9941 [2026-10-21 20:50:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2104 [2026-10-21 20:50:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3773 [2026-10-21 20:50:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0937 [2026-10-21 20:50:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1331 [2026-10-21 20:50:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5680 [2026-10-21 20:51:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0956 [2026-10-21 20:51:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0449 [2026-10-21 20:51:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3542 [2026-10-21 20:51:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0633 [2026-10-21 20:51:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8228 [2026-10-21 20:51:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0274 [2026-10-21 20:51:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7572 [2026-10-21 20:51:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7699 [2026-10-21 20:51:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0985 [2026-10-21 20:51:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3512 [2026-10-21 20:51:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8164 [2026-10-21 20:51:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6468 [2026-10-21 20:52:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8094 [2026-10-21 20:52:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8267 [2026-10-21 20:52:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8491 [2026-10-21 20:52:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8250 [2026-10-21 20:52:14][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,598 | MFU: 1.8% [2026-10-21 20:52:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7596 [2026-10-21 20:52:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6049 [2026-10-21 20:52:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8965 [2026-10-21 20:52:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6091 [2026-10-21 20:52:18][INFO][training.pretrain_engine] [Ph2|Step 353] loss=5.0508 | ppl=156.14 | aux=0.02506 | lr=2.99e-04 | gnorm=0.555 | tok/s=29,352 | MFU=7.0% | step=142896ms | tokens=1.172B | mem=4.2/32.7GB | cost=$28.94/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 07:43 (130.9h) | Phase3: SKIPPED | SFT: Oct 27 12:27 (135.6h) | DPO (Final): Oct 27 13:01 (136.2h) [2026-10-21 20:52:18][INFO][training.pretrain_engine] Starting global step 354 (accumulating 32 micro-batches)... [2026-10-21 20:52:18][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 20:52:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7458 [2026-10-21 20:52:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5540 [2026-10-21 20:52:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7860 [2026-10-21 20:52:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9618 [2026-10-21 20:52:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6516 [2026-10-21 20:52:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8522 [2026-10-21 20:52:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6426 [2026-10-21 20:52:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5823 [2026-10-21 20:53:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4225 [2026-10-21 20:53:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2601 [2026-10-21 20:53:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5498 [2026-10-21 20:53:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4973 [2026-10-21 20:53:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7686 [2026-10-21 20:53:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9069 [2026-10-21 20:53:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8228 [2026-10-21 20:53:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9838 [2026-10-21 20:53:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6191 [2026-10-21 20:53:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4222 [2026-10-21 20:53:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2261 [2026-10-21 20:53:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7740 [2026-10-21 20:54:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6868 [2026-10-21 20:54:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9568 [2026-10-21 20:54:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9152 [2026-10-21 20:54:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7859 [2026-10-21 20:54:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9661 [2026-10-21 20:54:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4038 [2026-10-21 20:54:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3177 [2026-10-21 20:54:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2148 [2026-10-21 20:54:36][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,597 | MFU: 1.8% [2026-10-21 20:54:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2766 [2026-10-21 20:54:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0343 [2026-10-21 20:54:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2360 [2026-10-21 20:54:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1616 [2026-10-21 20:54:41][INFO][training.pretrain_engine] [Ph2|Step 354] loss=5.0148 | ppl=150.62 | aux=0.02506 | lr=2.99e-04 | gnorm=0.562 | tok/s=29,352 | MFU=7.0% | step=142896ms | tokens=1.173B | mem=4.2/32.7GB | cost=$29.02/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 07:45 (130.9h) | Phase3: SKIPPED | SFT: Oct 27 12:29 (135.6h) | DPO (Final): Oct 27 13:03 (136.2h) [2026-10-21 20:54:41][INFO][training.pretrain_engine] Starting global step 355 (accumulating 32 micro-batches)... [2026-10-21 20:54:41][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 20:54:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4708 [2026-10-21 20:54:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3931 [2026-10-21 20:54:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3484 [2026-10-21 20:54:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5429 [2026-10-21 20:55:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8661 [2026-10-21 20:55:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4890 [2026-10-21 20:55:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3385 [2026-10-21 20:55:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4875 [2026-10-21 20:55:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9107 [2026-10-21 20:55:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8096 [2026-10-21 20:55:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3755 [2026-10-21 20:55:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9191 [2026-10-21 20:55:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1088 [2026-10-21 20:55:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=3.8767 [2026-10-21 20:55:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6443 [2026-10-21 20:55:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7390 [2026-10-21 20:56:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2136 [2026-10-21 20:56:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4463 [2026-10-21 20:56:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6452 [2026-10-21 20:56:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0084 [2026-10-21 20:56:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3567 [2026-10-21 20:56:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9946 [2026-10-21 20:56:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7180 [2026-10-21 20:56:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3469 [2026-10-21 20:56:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3165 [2026-10-21 20:56:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1682 [2026-10-21 20:56:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2732 [2026-10-21 20:56:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6576 [2026-10-21 20:56:59][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,599 | MFU: 1.8% [2026-10-21 20:57:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6026 [2026-10-21 20:57:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0622 [2026-10-21 20:57:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=3.8558 [2026-10-21 20:57:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2241 [2026-10-21 20:57:04][INFO][training.pretrain_engine] [Ph2|Step 355] loss=5.1014 | ppl=164.25 | aux=0.02507 | lr=2.99e-04 | gnorm=0.543 | tok/s=29,352 | MFU=7.0% | step=142898ms | tokens=1.174B | mem=4.2/32.7GB | cost=$29.10/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 07:47 (130.8h) | Phase3: SKIPPED | SFT: Oct 27 12:31 (135.6h) | DPO (Final): Oct 27 13:05 (136.1h) [2026-10-21 20:57:04][INFO][training.pretrain_engine] Starting global step 356 (accumulating 32 micro-batches)... [2026-10-21 20:57:04][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 20:57:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0034 [2026-10-21 20:57:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9879 [2026-10-21 20:57:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1806 [2026-10-21 20:57:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8413 [2026-10-21 20:57:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9913 [2026-10-21 20:57:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1827 [2026-10-21 20:57:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2857 [2026-10-21 20:57:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1757 [2026-10-21 20:57:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2408 [2026-10-21 20:57:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4183 [2026-10-21 20:57:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3623 [2026-10-21 20:57:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7545 [2026-10-21 20:58:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7679 [2026-10-21 20:58:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1500 [2026-10-21 20:58:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8593 [2026-10-21 20:58:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5814 [2026-10-21 20:58:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3372 [2026-10-21 20:58:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1313 [2026-10-21 20:58:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4729 [2026-10-21 20:58:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4005 [2026-10-21 20:58:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7252 [2026-10-21 20:58:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8993 [2026-10-21 20:58:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4301 [2026-10-21 20:58:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5191 [2026-10-21 20:59:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3686 [2026-10-21 20:59:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9652 [2026-10-21 20:59:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0911 [2026-10-21 20:59:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0612 [2026-10-21 20:59:22][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,599 | MFU: 1.8% [2026-10-21 20:59:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3368 [2026-10-21 20:59:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5333 [2026-10-21 20:59:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4289 [2026-10-21 20:59:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1279 [2026-10-21 20:59:27][INFO][training.pretrain_engine] [Ph2|Step 356] loss=5.1659 | ppl=175.20 | aux=0.02506 | lr=2.99e-04 | gnorm=0.504 | tok/s=29,351 | MFU=7.0% | step=142902ms | tokens=1.175B | mem=4.2/32.7GB | cost=$29.18/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 07:49 (130.8h) | Phase3: SKIPPED | SFT: Oct 27 12:33 (135.6h) | DPO (Final): Oct 27 13:07 (136.1h) [2026-10-21 20:59:27][INFO][training.pretrain_engine] Starting global step 357 (accumulating 32 micro-batches)... [2026-10-21 20:59:27][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 20:59:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0030 [2026-10-21 20:59:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0559 [2026-10-21 20:59:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3841 [2026-10-21 20:59:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7201 [2026-10-21 21:00:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3503 [2026-10-21 21:00:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2241 [2026-10-21 21:00:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7738 [2026-10-21 21:00:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4333 [2026-10-21 21:00:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3890 [2026-10-21 21:00:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0454 [2026-10-21 21:00:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9935 [2026-10-21 21:00:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3608 [2026-10-21 21:00:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1296 [2026-10-21 21:00:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5227 [2026-10-21 21:00:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5239 [2026-10-21 21:00:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7597 [2026-10-21 21:00:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3435 [2026-10-21 21:00:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7219 [2026-10-21 21:00:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.0975 [2026-10-21 21:00:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7818 [2026-10-21 21:01:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7433 [2026-10-21 21:01:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1027 [2026-10-21 21:01:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.0409 [2026-10-21 21:01:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.0757 [2026-10-21 21:01:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2255 [2026-10-21 21:01:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1860 [2026-10-21 21:01:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9210 [2026-10-21 21:01:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0839 [2026-10-21 21:01:45][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,583 | MFU: 1.8% [2026-10-21 21:01:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9669 [2026-10-21 21:01:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2010 [2026-10-21 21:01:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9706 [2026-10-21 21:01:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8488 [2026-10-21 21:01:49][INFO][training.pretrain_engine] [Ph2|Step 357] loss=5.0447 | ppl=155.20 | aux=0.02506 | lr=2.99e-04 | gnorm=0.465 | tok/s=29,351 | MFU=7.0% | step=142901ms | tokens=1.177B | mem=4.2/32.7GB | cost=$29.26/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 07:51 (130.8h) | Phase3: SKIPPED | SFT: Oct 27 12:35 (135.6h) | DPO (Final): Oct 27 13:09 (136.1h) [2026-10-21 21:01:49][INFO][training.pretrain_engine] Starting global step 358 (accumulating 32 micro-batches)... [2026-10-21 21:01:49][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 21:02:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2173 [2026-10-21 21:02:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8051 [2026-10-21 21:02:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0030 [2026-10-21 21:02:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9683 [2026-10-21 21:02:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3655 [2026-10-21 21:02:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1176 [2026-10-21 21:02:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9999 [2026-10-21 21:02:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5250 [2026-10-21 21:02:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1699 [2026-10-21 21:02:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1798 [2026-10-21 21:02:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2038 [2026-10-21 21:02:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3487 [2026-10-21 21:03:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9763 [2026-10-21 21:03:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1892 [2026-10-21 21:03:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3784 [2026-10-21 21:03:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9130 [2026-10-21 21:03:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6388 [2026-10-21 21:03:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2717 [2026-10-21 21:03:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8055 [2026-10-21 21:03:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5544 [2026-10-21 21:03:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7863 [2026-10-21 21:03:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7597 [2026-10-21 21:03:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8274 [2026-10-21 21:03:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4720 [2026-10-21 21:03:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1241 [2026-10-21 21:03:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5234 [2026-10-21 21:03:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8854 [2026-10-21 21:03:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9530 [2026-10-21 21:04:07][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,615 | MFU: 1.8% [2026-10-21 21:04:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2041 [2026-10-21 21:04:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0647 [2026-10-21 21:04:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3920 [2026-10-21 21:04:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2104 [2026-10-21 21:04:12][INFO][training.pretrain_engine] [Ph2|Step 358] loss=5.1446 | ppl=171.50 | aux=0.02506 | lr=2.99e-04 | gnorm=0.504 | tok/s=29,352 | MFU=7.0% | step=142899ms | tokens=1.178B | mem=4.2/32.7GB | cost=$29.34/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 07:52 (130.8h) | Phase3: SKIPPED | SFT: Oct 27 12:36 (135.5h) | DPO (Final): Oct 27 13:10 (136.1h) [2026-10-21 21:04:12][INFO][training.pretrain_engine] Starting global step 359 (accumulating 32 micro-batches)... [2026-10-21 21:04:12][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 21:04:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9438 [2026-10-21 21:04:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9418 [2026-10-21 21:04:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0719 [2026-10-21 21:04:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4395 [2026-10-21 21:04:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0251 [2026-10-21 21:04:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9446 [2026-10-21 21:04:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5137 [2026-10-21 21:04:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9774 [2026-10-21 21:05:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9122 [2026-10-21 21:05:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7590 [2026-10-21 21:05:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0970 [2026-10-21 21:05:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1079 [2026-10-21 21:05:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4626 [2026-10-21 21:05:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0215 [2026-10-21 21:05:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8036 [2026-10-21 21:05:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4452 [2026-10-21 21:05:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2772 [2026-10-21 21:05:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0717 [2026-10-21 21:05:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0484 [2026-10-21 21:05:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4667 [2026-10-21 21:05:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6392 [2026-10-21 21:05:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6367 [2026-10-21 21:05:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6818 [2026-10-21 21:05:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7215 [2026-10-21 21:06:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8765 [2026-10-21 21:06:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9591 [2026-10-21 21:06:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0192 [2026-10-21 21:06:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2002 [2026-10-21 21:06:29][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,619 | MFU: 1.8% [2026-10-21 21:06:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3379 [2026-10-21 21:06:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3461 [2026-10-21 21:06:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8467 [2026-10-21 21:06:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6147 [2026-10-21 21:06:35][INFO][training.pretrain_engine] [Ph2|Step 359] loss=5.0455 | ppl=155.32 | aux=0.02506 | lr=2.99e-04 | gnorm=0.484 | tok/s=29,351 | MFU=7.0% | step=142901ms | tokens=1.179B | mem=4.2/32.7GB | cost=$29.43/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 07:54 (130.8h) | Phase3: SKIPPED | SFT: Oct 27 12:38 (135.5h) | DPO (Final): Oct 27 13:12 (136.1h) [2026-10-21 21:06:35][INFO][training.pretrain_engine] Starting global step 360 (accumulating 32 micro-batches)... [2026-10-21 21:06:35][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 21:06:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3951 [2026-10-21 21:06:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2787 [2026-10-21 21:06:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1351 [2026-10-21 21:06:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8735 [2026-10-21 21:07:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7586 [2026-10-21 21:07:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1588 [2026-10-21 21:07:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3571 [2026-10-21 21:07:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6385 [2026-10-21 21:07:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3098 [2026-10-21 21:07:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6466 [2026-10-21 21:07:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3505 [2026-10-21 21:07:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1633 [2026-10-21 21:07:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0207 [2026-10-21 21:07:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7603 [2026-10-21 21:07:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5681 [2026-10-21 21:07:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7020 [2026-10-21 21:08:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4679 [2026-10-21 21:08:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0414 [2026-10-21 21:08:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2733 [2026-10-21 21:08:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0924 [2026-10-21 21:08:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3766 [2026-10-21 21:08:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9190 [2026-10-21 21:08:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9628 [2026-10-21 21:08:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6895 [2026-10-21 21:08:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2910 [2026-10-21 21:08:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1081 [2026-10-21 21:08:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9239 [2026-10-21 21:08:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0988 [2026-10-21 21:08:52][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,615 | MFU: 1.8% [2026-10-21 21:08:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0302 [2026-10-21 21:08:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9652 [2026-10-21 21:08:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4899 [2026-10-21 21:08:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9333 [2026-10-21 21:08:58][INFO][training.pretrain_engine] [Ph2|Step 360] loss=5.1120 | ppl=166.00 | aux=0.02506 | lr=2.99e-04 | gnorm=0.434 | tok/s=29,350 | MFU=7.0% | step=142908ms | tokens=1.180B | mem=4.2/32.7GB | cost=$29.51/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 07:57 (130.8h) | Phase3: SKIPPED | SFT: Oct 27 12:40 (135.5h) | DPO (Final): Oct 27 13:15 (136.1h) [2026-10-21 21:09:01][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step360 | Tokens: 1.180B | Spent: $29.51 | Remaining: $970.49 [2026-10-21 21:09:01][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step360 [2026-10-21 21:09:01][INFO][training.pretrain_engine] Starting global step 361 (accumulating 32 micro-batches)... [2026-10-21 21:09:01][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 21:09:01][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 21:09:01][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 21:09:01][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 21:09:01][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step360 to Phase-Technologies/forge-500m-pretrain (path: phase2_step360)... [2026-10-21 21:09:01][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 21:09:11][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 21:09:11][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step360 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 21:09:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9751 [2026-10-21 21:09:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4366 [2026-10-21 21:09:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6097 [2026-10-21 21:09:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6980 [2026-10-21 21:09:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9007 [2026-10-21 21:09:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9389 [2026-10-21 21:09:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5621 [2026-10-21 21:09:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1441 [2026-10-21 21:09:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1078 [2026-10-21 21:09:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.0788 [2026-10-21 21:09:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0474 [2026-10-21 21:09:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7631 [2026-10-21 21:10:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7907 [2026-10-21 21:10:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6413 [2026-10-21 21:10:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8422 [2026-10-21 21:10:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8154 [2026-10-21 21:10:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0459 [2026-10-21 21:10:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2908 [2026-10-21 21:10:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3194 [2026-10-21 21:10:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5597 [2026-10-21 21:10:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0260 [2026-10-21 21:10:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7680 [2026-10-21 21:10:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3270 [2026-10-21 21:10:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1723 [2026-10-21 21:11:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0469 [2026-10-21 21:11:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9511 [2026-10-21 21:11:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1257 [2026-10-21 21:11:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2237 [2026-10-21 21:11:19][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,592 | MFU: 1.8% [2026-10-21 21:11:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1828 [2026-10-21 21:11:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1530 [2026-10-21 21:11:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5995 [2026-10-21 21:11:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4319 [2026-10-21 21:11:23][INFO][training.pretrain_engine] [Ph2|Step 361] loss=5.1510 | ppl=172.60 | aux=0.02507 | lr=2.99e-04 | gnorm=0.457 | tok/s=29,349 | MFU=7.0% | step=142910ms | tokens=1.181B | mem=4.2/32.7GB | cost=$29.59/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 07:58 (130.8h) | Phase3: SKIPPED | SFT: Oct 27 12:42 (135.5h) | DPO (Final): Oct 27 13:17 (136.1h) [2026-10-21 21:11:23][INFO][training.pretrain_engine] Starting global step 362 (accumulating 32 micro-batches)... [2026-10-21 21:11:23][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 21:11:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0592 [2026-10-21 21:11:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2475 [2026-10-21 21:11:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6546 [2026-10-21 21:11:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4632 [2026-10-21 21:11:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3482 [2026-10-21 21:11:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3659 [2026-10-21 21:11:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8809 [2026-10-21 21:11:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3200 [2026-10-21 21:12:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9052 [2026-10-21 21:12:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0066 [2026-10-21 21:12:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1814 [2026-10-21 21:12:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9121 [2026-10-21 21:12:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6910 [2026-10-21 21:12:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6371 [2026-10-21 21:12:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7721 [2026-10-21 21:12:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4423 [2026-10-21 21:12:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5295 [2026-10-21 21:12:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0721 [2026-10-21 21:12:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1433 [2026-10-21 21:12:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3529 [2026-10-21 21:13:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9385 [2026-10-21 21:13:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8663 [2026-10-21 21:13:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1850 [2026-10-21 21:13:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9321 [2026-10-21 21:13:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9002 [2026-10-21 21:13:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2516 [2026-10-21 21:13:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0647 [2026-10-21 21:13:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2481 [2026-10-21 21:13:41][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,612 | MFU: 1.8% [2026-10-21 21:13:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1687 [2026-10-21 21:13:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6693 [2026-10-21 21:13:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2878 [2026-10-21 21:13:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5715 [2026-10-21 21:13:47][INFO][training.pretrain_engine] [Ph2|Step 362] loss=5.0527 | ppl=156.44 | aux=0.02506 | lr=2.99e-04 | gnorm=0.453 | tok/s=29,348 | MFU=7.0% | step=142917ms | tokens=1.182B | mem=4.2/32.7GB | cost=$29.67/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 08:01 (130.8h) | Phase3: SKIPPED | SFT: Oct 27 12:45 (135.5h) | DPO (Final): Oct 27 13:19 (136.1h) [2026-10-21 21:13:47][INFO][training.pretrain_engine] Starting global step 363 (accumulating 32 micro-batches)... [2026-10-21 21:13:47][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 21:14:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1245 [2026-10-21 21:14:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0695 [2026-10-21 21:14:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2843 [2026-10-21 21:14:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9497 [2026-10-21 21:14:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8600 [2026-10-21 21:14:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2465 [2026-10-21 21:14:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4105 [2026-10-21 21:14:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4103 [2026-10-21 21:14:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4665 [2026-10-21 21:14:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2335 [2026-10-21 21:14:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8136 [2026-10-21 21:14:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4468 [2026-10-21 21:14:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1854 [2026-10-21 21:14:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8089 [2026-10-21 21:14:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2938 [2026-10-21 21:14:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8460 [2026-10-21 21:15:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9389 [2026-10-21 21:15:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1856 [2026-10-21 21:15:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8914 [2026-10-21 21:15:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0404 [2026-10-21 21:15:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1548 [2026-10-21 21:15:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0336 [2026-10-21 21:15:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1216 [2026-10-21 21:15:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0466 [2026-10-21 21:15:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5807 [2026-10-21 21:15:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1382 [2026-10-21 21:15:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1355 [2026-10-21 21:15:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2454 [2026-10-21 21:16:05][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,606 | MFU: 1.8% [2026-10-21 21:16:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0638 [2026-10-21 21:16:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6701 [2026-10-21 21:16:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1574 [2026-10-21 21:16:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9979 [2026-10-21 21:16:11][INFO][training.pretrain_engine] [Ph2|Step 363] loss=5.0717 | ppl=159.45 | aux=0.02505 | lr=2.99e-04 | gnorm=0.504 | tok/s=29,345 | MFU=7.0% | step=142932ms | tokens=1.183B | mem=4.2/32.7GB | cost=$29.76/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 08:03 (130.8h) | Phase3: SKIPPED | SFT: Oct 27 12:47 (135.5h) | DPO (Final): Oct 27 13:21 (136.1h) [2026-10-21 21:16:11][INFO][training.pretrain_engine] Starting global step 364 (accumulating 32 micro-batches)... [2026-10-21 21:16:11][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 21:16:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0020 [2026-10-21 21:16:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1941 [2026-10-21 21:16:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9440 [2026-10-21 21:16:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8745 [2026-10-21 21:16:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9936 [2026-10-21 21:16:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6994 [2026-10-21 21:16:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6134 [2026-10-21 21:16:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7996 [2026-10-21 21:17:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1771 [2026-10-21 21:17:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8385 [2026-10-21 21:17:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6011 [2026-10-21 21:17:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0282 [2026-10-21 21:17:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1838 [2026-10-21 21:17:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1622 [2026-10-21 21:17:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1723 [2026-10-21 21:17:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8951 [2026-10-21 21:17:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1484 [2026-10-21 21:17:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2069 [2026-10-21 21:17:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7256 [2026-10-21 21:17:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3237 [2026-10-21 21:17:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8736 [2026-10-21 21:17:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0298 [2026-10-21 21:17:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8347 [2026-10-21 21:17:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9868 [2026-10-21 21:18:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3800 [2026-10-21 21:18:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3243 [2026-10-21 21:18:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4501 [2026-10-21 21:18:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4938 [2026-10-21 21:18:28][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,616 | MFU: 1.8% [2026-10-21 21:18:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2064 [2026-10-21 21:18:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9923 [2026-10-21 21:18:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1369 [2026-10-21 21:18:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4475 [2026-10-21 21:18:34][INFO][training.pretrain_engine] [Ph2|Step 364] loss=5.0600 | ppl=157.60 | aux=0.02507 | lr=2.99e-04 | gnorm=0.527 | tok/s=29,345 | MFU=7.0% | step=142932ms | tokens=1.184B | mem=4.2/32.7GB | cost=$29.84/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 08:05 (130.8h) | Phase3: SKIPPED | SFT: Oct 27 12:49 (135.5h) | DPO (Final): Oct 27 13:23 (136.1h) [2026-10-21 21:18:34][INFO][training.pretrain_engine] Starting global step 365 (accumulating 32 micro-batches)... [2026-10-21 21:18:34][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 21:18:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7369 [2026-10-21 21:18:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4786 [2026-10-21 21:18:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4143 [2026-10-21 21:18:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2214 [2026-10-21 21:19:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1824 [2026-10-21 21:19:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4900 [2026-10-21 21:19:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0886 [2026-10-21 21:19:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4448 [2026-10-21 21:19:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0259 [2026-10-21 21:19:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1572 [2026-10-21 21:19:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2543 [2026-10-21 21:19:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4446 [2026-10-21 21:19:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7350 [2026-10-21 21:19:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2129 [2026-10-21 21:19:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0383 [2026-10-21 21:19:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6735 [2026-10-21 21:20:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4255 [2026-10-21 21:20:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7332 [2026-10-21 21:20:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1671 [2026-10-21 21:20:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1120 [2026-10-21 21:20:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3289 [2026-10-21 21:20:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4302 [2026-10-21 21:20:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4888 [2026-10-21 21:20:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2502 [2026-10-21 21:20:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3295 [2026-10-21 21:20:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0344 [2026-10-21 21:20:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7361 [2026-10-21 21:20:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2560 [2026-10-21 21:20:52][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,606 | MFU: 1.8% [2026-10-21 21:20:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0999 [2026-10-21 21:20:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2643 [2026-10-21 21:20:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8652 [2026-10-21 21:20:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2307 [2026-10-21 21:20:56][INFO][training.pretrain_engine] [Ph2|Step 365] loss=5.1046 | ppl=164.78 | aux=0.02506 | lr=2.99e-04 | gnorm=0.484 | tok/s=29,345 | MFU=7.0% | step=142933ms | tokens=1.185B | mem=4.2/32.7GB | cost=$29.92/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 08:07 (130.8h) | Phase3: SKIPPED | SFT: Oct 27 12:51 (135.5h) | DPO (Final): Oct 27 13:25 (136.1h) [2026-10-21 21:20:56][INFO][training.pretrain_engine] Starting global step 366 (accumulating 32 micro-batches)... [2026-10-21 21:20:56][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 21:21:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7357 [2026-10-21 21:21:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2384 [2026-10-21 21:21:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0690 [2026-10-21 21:21:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1044 [2026-10-21 21:21:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3706 [2026-10-21 21:21:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.0041 [2026-10-21 21:21:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6747 [2026-10-21 21:21:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6448 [2026-10-21 21:21:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3155 [2026-10-21 21:21:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5537 [2026-10-21 21:21:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6687 [2026-10-21 21:21:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1821 [2026-10-21 21:22:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1624 [2026-10-21 21:22:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2024 [2026-10-21 21:22:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6332 [2026-10-21 21:22:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2704 [2026-10-21 21:22:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5015 [2026-10-21 21:22:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3712 [2026-10-21 21:22:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0660 [2026-10-21 21:22:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1926 [2026-10-21 21:22:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3864 [2026-10-21 21:22:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1779 [2026-10-21 21:22:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8289 [2026-10-21 21:22:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5292 [2026-10-21 21:23:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7998 [2026-10-21 21:23:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9882 [2026-10-21 21:23:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2017 [2026-10-21 21:23:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6847 [2026-10-21 21:23:14][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,620 | MFU: 1.8% [2026-10-21 21:23:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6937 [2026-10-21 21:23:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9899 [2026-10-21 21:23:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5689 [2026-10-21 21:23:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8608 [2026-10-21 21:23:19][INFO][training.pretrain_engine] [Ph2|Step 366] loss=5.1972 | ppl=180.77 | aux=0.02507 | lr=2.99e-04 | gnorm=0.453 | tok/s=29,344 | MFU=7.0% | step=142935ms | tokens=1.186B | mem=4.2/32.7GB | cost=$30.00/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 08:09 (130.8h) | Phase3: SKIPPED | SFT: Oct 27 12:53 (135.5h) | DPO (Final): Oct 27 13:27 (136.1h) [2026-10-21 21:23:19][INFO][training.pretrain_engine] Starting global step 367 (accumulating 32 micro-batches)... [2026-10-21 21:23:19][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 21:23:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4085 [2026-10-21 21:23:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8944 [2026-10-21 21:23:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6215 [2026-10-21 21:23:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0711 [2026-10-21 21:23:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9003 [2026-10-21 21:23:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0405 [2026-10-21 21:23:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0977 [2026-10-21 21:23:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5462 [2026-10-21 21:24:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2928 [2026-10-21 21:24:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4819 [2026-10-21 21:24:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9504 [2026-10-21 21:24:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1862 [2026-10-21 21:24:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1001 [2026-10-21 21:24:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6178 [2026-10-21 21:24:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7802 [2026-10-21 21:24:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9212 [2026-10-21 21:24:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4568 [2026-10-21 21:24:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8222 [2026-10-21 21:24:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3651 [2026-10-21 21:24:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0938 [2026-10-21 21:25:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5755 [2026-10-21 21:25:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9385 [2026-10-21 21:25:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2367 [2026-10-21 21:25:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9753 [2026-10-21 21:25:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6431 [2026-10-21 21:25:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8148 [2026-10-21 21:25:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8137 [2026-10-21 21:25:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7557 [2026-10-21 21:25:37][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,621 | MFU: 1.8% [2026-10-21 21:25:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2266 [2026-10-21 21:25:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8408 [2026-10-21 21:25:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8336 [2026-10-21 21:25:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3518 [2026-10-21 21:25:42][INFO][training.pretrain_engine] [Ph2|Step 367] loss=5.1579 | ppl=173.81 | aux=0.02506 | lr=2.99e-04 | gnorm=0.482 | tok/s=29,345 | MFU=7.0% | step=142933ms | tokens=1.187B | mem=4.2/32.7GB | cost=$30.08/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 08:10 (130.8h) | Phase3: SKIPPED | SFT: Oct 27 12:54 (135.5h) | DPO (Final): Oct 27 13:29 (136.1h) [2026-10-21 21:25:42][INFO][training.pretrain_engine] Starting global step 368 (accumulating 32 micro-batches)... [2026-10-21 21:25:42][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 21:25:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8214 [2026-10-21 21:25:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6150 [2026-10-21 21:25:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5726 [2026-10-21 21:26:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2447 [2026-10-21 21:26:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2495 [2026-10-21 21:26:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3906 [2026-10-21 21:26:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3021 [2026-10-21 21:26:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9379 [2026-10-21 21:26:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6830 [2026-10-21 21:26:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2964 [2026-10-21 21:26:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0539 [2026-10-21 21:26:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9739 [2026-10-21 21:26:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1793 [2026-10-21 21:26:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3922 [2026-10-21 21:26:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2320 [2026-10-21 21:26:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5599 [2026-10-21 21:27:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9796 [2026-10-21 21:27:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8857 [2026-10-21 21:27:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9756 [2026-10-21 21:27:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9380 [2026-10-21 21:27:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9443 [2026-10-21 21:27:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0531 [2026-10-21 21:27:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2590 [2026-10-21 21:27:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9434 [2026-10-21 21:27:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2867 [2026-10-21 21:27:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9889 [2026-10-21 21:27:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2180 [2026-10-21 21:27:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0423 [2026-10-21 21:27:59][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,615 | MFU: 1.8% [2026-10-21 21:28:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7810 [2026-10-21 21:28:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2491 [2026-10-21 21:28:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3698 [2026-10-21 21:28:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4817 [2026-10-21 21:28:04][INFO][training.pretrain_engine] [Ph2|Step 368] loss=5.1042 | ppl=164.72 | aux=0.02505 | lr=2.99e-04 | gnorm=0.512 | tok/s=29,345 | MFU=7.0% | step=142931ms | tokens=1.188B | mem=4.2/32.7GB | cost=$30.16/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 08:12 (130.7h) | Phase3: SKIPPED | SFT: Oct 27 12:56 (135.5h) | DPO (Final): Oct 27 13:30 (136.0h) [2026-10-21 21:28:04][INFO][training.pretrain_engine] Starting global step 369 (accumulating 32 micro-batches)... [2026-10-21 21:28:04][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 21:28:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3979 [2026-10-21 21:28:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2205 [2026-10-21 21:28:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2835 [2026-10-21 21:28:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6603 [2026-10-21 21:28:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8241 [2026-10-21 21:28:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1654 [2026-10-21 21:28:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2692 [2026-10-21 21:28:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2589 [2026-10-21 21:28:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3402 [2026-10-21 21:28:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8835 [2026-10-21 21:28:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0949 [2026-10-21 21:28:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3626 [2026-10-21 21:29:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6132 [2026-10-21 21:29:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.0845 [2026-10-21 21:29:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2777 [2026-10-21 21:29:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1454 [2026-10-21 21:29:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5904 [2026-10-21 21:29:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0415 [2026-10-21 21:29:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9217 [2026-10-21 21:29:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3511 [2026-10-21 21:29:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8036 [2026-10-21 21:29:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7627 [2026-10-21 21:29:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1432 [2026-10-21 21:29:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3135 [2026-10-21 21:30:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3606 [2026-10-21 21:30:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6705 [2026-10-21 21:30:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6931 [2026-10-21 21:30:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7122 [2026-10-21 21:30:22][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,613 | MFU: 1.8% [2026-10-21 21:30:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3200 [2026-10-21 21:30:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1406 [2026-10-21 21:30:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2814 [2026-10-21 21:30:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0208 [2026-10-21 21:30:27][INFO][training.pretrain_engine] [Ph2|Step 369] loss=4.9721 | ppl=144.33 | aux=0.02507 | lr=2.98e-04 | gnorm=0.570 | tok/s=29,345 | MFU=7.0% | step=142929ms | tokens=1.189B | mem=4.2/32.7GB | cost=$30.25/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 08:14 (130.7h) | Phase3: SKIPPED | SFT: Oct 27 12:58 (135.5h) | DPO (Final): Oct 27 13:32 (136.0h) [2026-10-21 21:30:31][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step369 | Tokens: 1.189B | Spent: $30.25 | Remaining: $969.75 [2026-10-21 21:30:31][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step369 [2026-10-21 21:30:31][INFO][training.pretrain_engine] Starting global step 370 (accumulating 32 micro-batches)... [2026-10-21 21:30:31][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 21:30:31][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 21:30:31][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 21:30:31][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 21:30:31][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step369 to Phase-Technologies/forge-500m-pretrain (path: phase2_step369)... [2026-10-21 21:30:31][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 21:30:40][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 21:30:40][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step369 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 21:30:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4746 [2026-10-21 21:30:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8357 [2026-10-21 21:30:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1342 [2026-10-21 21:30:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3730 [2026-10-21 21:31:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9236 [2026-10-21 21:31:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2817 [2026-10-21 21:31:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5530 [2026-10-21 21:31:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9784 [2026-10-21 21:31:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2069 [2026-10-21 21:31:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0896 [2026-10-21 21:31:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6500 [2026-10-21 21:31:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9144 [2026-10-21 21:31:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2062 [2026-10-21 21:31:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4844 [2026-10-21 21:31:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8117 [2026-10-21 21:31:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6018 [2026-10-21 21:31:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3128 [2026-10-21 21:31:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2912 [2026-10-21 21:31:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2591 [2026-10-21 21:31:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2800 [2026-10-21 21:32:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1584 [2026-10-21 21:32:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8751 [2026-10-21 21:32:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2527 [2026-10-21 21:32:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4078 [2026-10-21 21:32:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4240 [2026-10-21 21:32:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1462 [2026-10-21 21:32:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7023 [2026-10-21 21:32:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2228 [2026-10-21 21:32:48][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,619 | MFU: 1.8% [2026-10-21 21:32:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0679 [2026-10-21 21:32:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0616 [2026-10-21 21:32:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4479 [2026-10-21 21:32:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4164 [2026-10-21 21:32:53][INFO][training.pretrain_engine] [Ph2|Step 370] loss=5.1268 | ppl=168.48 | aux=0.02507 | lr=2.98e-04 | gnorm=0.605 | tok/s=29,347 | MFU=7.0% | step=142922ms | tokens=1.190B | mem=4.2/32.7GB | cost=$30.33/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 08:15 (130.7h) | Phase3: SKIPPED | SFT: Oct 27 12:59 (135.4h) | DPO (Final): Oct 27 13:33 (136.0h) [2026-10-21 21:32:53][INFO][training.pretrain_engine] Starting global step 371 (accumulating 32 micro-batches)... [2026-10-21 21:32:53][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 21:33:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8366 [2026-10-21 21:33:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8658 [2026-10-21 21:33:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7847 [2026-10-21 21:33:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0175 [2026-10-21 21:33:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0904 [2026-10-21 21:33:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2233 [2026-10-21 21:33:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0098 [2026-10-21 21:33:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2522 [2026-10-21 21:33:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3833 [2026-10-21 21:33:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1633 [2026-10-21 21:33:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2001 [2026-10-21 21:33:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8949 [2026-10-21 21:34:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0053 [2026-10-21 21:34:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5019 [2026-10-21 21:34:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9867 [2026-10-21 21:34:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1158 [2026-10-21 21:34:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9941 [2026-10-21 21:34:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2036 [2026-10-21 21:34:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2467 [2026-10-21 21:34:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5409 [2026-10-21 21:34:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0913 [2026-10-21 21:34:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1225 [2026-10-21 21:34:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2705 [2026-10-21 21:34:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0394 [2026-10-21 21:34:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3954 [2026-10-21 21:34:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8782 [2026-10-21 21:34:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2935 [2026-10-21 21:34:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7661 [2026-10-21 21:35:11][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,606 | MFU: 1.8% [2026-10-21 21:35:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0043 [2026-10-21 21:35:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3064 [2026-10-21 21:35:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6363 [2026-10-21 21:35:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.3161 [2026-10-21 21:35:15][INFO][training.pretrain_engine] [Ph2|Step 371] loss=5.1142 | ppl=166.37 | aux=0.02507 | lr=2.98e-04 | gnorm=0.688 | tok/s=29,346 | MFU=7.0% | step=142924ms | tokens=1.191B | mem=4.2/32.7GB | cost=$30.41/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 08:17 (130.7h) | Phase3: SKIPPED | SFT: Oct 27 13:01 (135.4h) | DPO (Final): Oct 27 13:35 (136.0h) [2026-10-21 21:35:15][INFO][training.pretrain_engine] Starting global step 372 (accumulating 32 micro-batches)... [2026-10-21 21:35:15][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 21:35:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3634 [2026-10-21 21:35:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9773 [2026-10-21 21:35:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5786 [2026-10-21 21:35:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3781 [2026-10-21 21:35:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1259 [2026-10-21 21:35:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2832 [2026-10-21 21:35:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9185 [2026-10-21 21:35:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0949 [2026-10-21 21:36:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3720 [2026-10-21 21:36:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7479 [2026-10-21 21:36:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8477 [2026-10-21 21:36:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8821 [2026-10-21 21:36:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8264 [2026-10-21 21:36:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7173 [2026-10-21 21:36:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2902 [2026-10-21 21:36:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0300 [2026-10-21 21:36:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0729 [2026-10-21 21:36:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6289 [2026-10-21 21:36:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8081 [2026-10-21 21:36:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4222 [2026-10-21 21:37:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6445 [2026-10-21 21:37:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2125 [2026-10-21 21:37:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0200 [2026-10-21 21:37:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8229 [2026-10-21 21:37:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1211 [2026-10-21 21:37:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8833 [2026-10-21 21:37:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9622 [2026-10-21 21:37:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1238 [2026-10-21 21:37:33][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,620 | MFU: 1.8% [2026-10-21 21:37:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0809 [2026-10-21 21:37:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2477 [2026-10-21 21:37:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4371 [2026-10-21 21:37:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9408 [2026-10-21 21:37:38][INFO][training.pretrain_engine] [Ph2|Step 372] loss=4.9513 | ppl=141.36 | aux=0.02506 | lr=2.98e-04 | gnorm=0.680 | tok/s=29,348 | MFU=7.0% | step=142917ms | tokens=1.192B | mem=4.2/32.7GB | cost=$30.49/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 08:19 (130.7h) | Phase3: SKIPPED | SFT: Oct 27 13:03 (135.4h) | DPO (Final): Oct 27 13:37 (136.0h) [2026-10-21 21:37:38][INFO][training.pretrain_engine] Starting global step 373 (accumulating 32 micro-batches)... [2026-10-21 21:37:38][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 21:37:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4771 [2026-10-21 21:37:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9162 [2026-10-21 21:37:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0783 [2026-10-21 21:37:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7761 [2026-10-21 21:38:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8740 [2026-10-21 21:38:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7933 [2026-10-21 21:38:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5104 [2026-10-21 21:38:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0684 [2026-10-21 21:38:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1691 [2026-10-21 21:38:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3644 [2026-10-21 21:38:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1258 [2026-10-21 21:38:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7161 [2026-10-21 21:38:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0093 [2026-10-21 21:38:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1238 [2026-10-21 21:38:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4779 [2026-10-21 21:38:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0535 [2026-10-21 21:39:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9704 [2026-10-21 21:39:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0824 [2026-10-21 21:39:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9893 [2026-10-21 21:39:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6582 [2026-10-21 21:39:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7781 [2026-10-21 21:39:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9333 [2026-10-21 21:39:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0691 [2026-10-21 21:39:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8564 [2026-10-21 21:39:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0127 [2026-10-21 21:39:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3754 [2026-10-21 21:39:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2006 [2026-10-21 21:39:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4339 [2026-10-21 21:39:56][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,620 | MFU: 1.8% [2026-10-21 21:40:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5404 [2026-10-21 21:40:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6130 [2026-10-21 21:40:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8767 [2026-10-21 21:40:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9617 [2026-10-21 21:40:01][INFO][training.pretrain_engine] [Ph2|Step 373] loss=5.1161 | ppl=166.69 | aux=0.02506 | lr=2.98e-04 | gnorm=0.719 | tok/s=29,349 | MFU=7.0% | step=142913ms | tokens=1.193B | mem=4.2/32.7GB | cost=$30.57/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 08:20 (130.7h) | Phase3: SKIPPED | SFT: Oct 27 13:04 (135.4h) | DPO (Final): Oct 27 13:38 (136.0h) [2026-10-21 21:40:01][INFO][training.pretrain_engine] Starting global step 374 (accumulating 32 micro-batches)... [2026-10-21 21:40:01][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 21:40:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8281 [2026-10-21 21:40:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0763 [2026-10-21 21:40:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1528 [2026-10-21 21:40:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9389 [2026-10-21 21:40:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7766 [2026-10-21 21:40:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8988 [2026-10-21 21:40:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0749 [2026-10-21 21:40:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9626 [2026-10-21 21:40:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2633 [2026-10-21 21:40:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2811 [2026-10-21 21:40:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5604 [2026-10-21 21:40:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=3.9803 [2026-10-21 21:41:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1797 [2026-10-21 21:41:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1972 [2026-10-21 21:41:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3933 [2026-10-21 21:41:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8795 [2026-10-21 21:41:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0270 [2026-10-21 21:41:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4496 [2026-10-21 21:41:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4997 [2026-10-21 21:41:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6585 [2026-10-21 21:41:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5433 [2026-10-21 21:41:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8070 [2026-10-21 21:41:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9956 [2026-10-21 21:41:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1184 [2026-10-21 21:42:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4894 [2026-10-21 21:42:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0524 [2026-10-21 21:42:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7705 [2026-10-21 21:42:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3283 [2026-10-21 21:42:19][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,604 | MFU: 1.8% [2026-10-21 21:42:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5499 [2026-10-21 21:42:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2820 [2026-10-21 21:42:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2167 [2026-10-21 21:42:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4172 [2026-10-21 21:42:24][INFO][training.pretrain_engine] [Ph2|Step 374] loss=5.1584 | ppl=173.88 | aux=0.02506 | lr=2.98e-04 | gnorm=0.785 | tok/s=29,349 | MFU=7.0% | step=142909ms | tokens=1.194B | mem=4.2/32.7GB | cost=$30.66/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 08:22 (130.7h) | Phase3: SKIPPED | SFT: Oct 27 13:06 (135.4h) | DPO (Final): Oct 27 13:40 (136.0h) [2026-10-21 21:42:24][INFO][training.pretrain_engine] Starting global step 375 (accumulating 32 micro-batches)... [2026-10-21 21:42:24][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 21:42:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6508 [2026-10-21 21:42:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5712 [2026-10-21 21:42:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0057 [2026-10-21 21:42:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3428 [2026-10-21 21:42:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9334 [2026-10-21 21:42:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0164 [2026-10-21 21:42:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9395 [2026-10-21 21:42:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5802 [2026-10-21 21:43:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8084 [2026-10-21 21:43:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5777 [2026-10-21 21:43:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=3.9885 [2026-10-21 21:43:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9999 [2026-10-21 21:43:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3195 [2026-10-21 21:43:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1487 [2026-10-21 21:43:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4689 [2026-10-21 21:43:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0730 [2026-10-21 21:43:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3080 [2026-10-21 21:43:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8340 [2026-10-21 21:43:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4146 [2026-10-21 21:43:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4543 [2026-10-21 21:44:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0681 [2026-10-21 21:44:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0267 [2026-10-21 21:44:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9834 [2026-10-21 21:44:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0831 [2026-10-21 21:44:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3657 [2026-10-21 21:44:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.1210 [2026-10-21 21:44:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9286 [2026-10-21 21:44:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4990 [2026-10-21 21:44:41][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,608 | MFU: 1.8% [2026-10-21 21:44:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1294 [2026-10-21 21:44:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7161 [2026-10-21 21:44:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1607 [2026-10-21 21:44:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1005 [2026-10-21 21:44:46][INFO][training.pretrain_engine] [Ph2|Step 375] loss=5.0328 | ppl=153.37 | aux=0.02506 | lr=2.98e-04 | gnorm=0.863 | tok/s=29,351 | MFU=7.0% | step=142903ms | tokens=1.195B | mem=4.2/32.7GB | cost=$30.74/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 08:23 (130.6h) | Phase3: SKIPPED | SFT: Oct 27 13:07 (135.4h) | DPO (Final): Oct 27 13:41 (135.9h) [2026-10-21 21:44:46][INFO][training.pretrain_engine] Starting global step 376 (accumulating 32 micro-batches)... [2026-10-21 21:44:46][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 21:45:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8717 [2026-10-21 21:45:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6405 [2026-10-21 21:45:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4136 [2026-10-21 21:45:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3977 [2026-10-21 21:45:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3680 [2026-10-21 21:45:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1853 [2026-10-21 21:45:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1221 [2026-10-21 21:45:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4555 [2026-10-21 21:45:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4974 [2026-10-21 21:45:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6569 [2026-10-21 21:45:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9617 [2026-10-21 21:45:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7806 [2026-10-21 21:45:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4477 [2026-10-21 21:45:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6487 [2026-10-21 21:45:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7272 [2026-10-21 21:45:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3860 [2026-10-21 21:46:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7492 [2026-10-21 21:46:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3592 [2026-10-21 21:46:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2740 [2026-10-21 21:46:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0251 [2026-10-21 21:46:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0237 [2026-10-21 21:46:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0114 [2026-10-21 21:46:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8569 [2026-10-21 21:46:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8655 [2026-10-21 21:46:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8966 [2026-10-21 21:46:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0097 [2026-10-21 21:46:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7580 [2026-10-21 21:46:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2917 [2026-10-21 21:47:04][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,593 | MFU: 1.8% [2026-10-21 21:47:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2417 [2026-10-21 21:47:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9789 [2026-10-21 21:47:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1591 [2026-10-21 21:47:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2224 [2026-10-21 21:47:09][INFO][training.pretrain_engine] [Ph2|Step 376] loss=4.9390 | ppl=139.63 | aux=0.02506 | lr=2.98e-04 | gnorm=0.793 | tok/s=29,351 | MFU=7.0% | step=142904ms | tokens=1.196B | mem=4.2/32.7GB | cost=$30.82/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 08:25 (130.6h) | Phase3: SKIPPED | SFT: Oct 27 13:09 (135.4h) | DPO (Final): Oct 27 13:43 (135.9h) [2026-10-21 21:47:09][INFO][training.pretrain_engine] Starting global step 377 (accumulating 32 micro-batches)... [2026-10-21 21:47:09][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 21:47:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2003 [2026-10-21 21:47:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9997 [2026-10-21 21:47:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0722 [2026-10-21 21:47:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3462 [2026-10-21 21:47:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2781 [2026-10-21 21:47:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3952 [2026-10-21 21:47:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3138 [2026-10-21 21:47:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9450 [2026-10-21 21:48:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5798 [2026-10-21 21:48:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8110 [2026-10-21 21:48:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8924 [2026-10-21 21:48:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8032 [2026-10-21 21:48:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6655 [2026-10-21 21:48:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0615 [2026-10-21 21:48:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1307 [2026-10-21 21:48:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1738 [2026-10-21 21:48:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0877 [2026-10-21 21:48:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2663 [2026-10-21 21:48:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3028 [2026-10-21 21:48:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7830 [2026-10-21 21:48:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3382 [2026-10-21 21:48:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0377 [2026-10-21 21:48:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7927 [2026-10-21 21:48:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0809 [2026-10-21 21:49:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6780 [2026-10-21 21:49:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2784 [2026-10-21 21:49:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4852 [2026-10-21 21:49:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6076 [2026-10-21 21:49:27][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,603 | MFU: 1.8% [2026-10-21 21:49:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2330 [2026-10-21 21:49:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9648 [2026-10-21 21:49:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7130 [2026-10-21 21:49:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8090 [2026-10-21 21:49:31][INFO][training.pretrain_engine] [Ph2|Step 377] loss=5.1475 | ppl=172.00 | aux=0.02507 | lr=2.98e-04 | gnorm=0.824 | tok/s=29,352 | MFU=7.0% | step=142897ms | tokens=1.197B | mem=4.2/32.7GB | cost=$30.90/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 08:26 (130.6h) | Phase3: SKIPPED | SFT: Oct 27 13:10 (135.4h) | DPO (Final): Oct 27 13:44 (135.9h) [2026-10-21 21:49:31][INFO][training.pretrain_engine] Starting global step 378 (accumulating 32 micro-batches)... [2026-10-21 21:49:31][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 21:49:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5049 [2026-10-21 21:49:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4957 [2026-10-21 21:49:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7839 [2026-10-21 21:49:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6543 [2026-10-21 21:50:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0621 [2026-10-21 21:50:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0774 [2026-10-21 21:50:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1522 [2026-10-21 21:50:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8823 [2026-10-21 21:50:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3479 [2026-10-21 21:50:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7414 [2026-10-21 21:50:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7619 [2026-10-21 21:50:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1441 [2026-10-21 21:50:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2368 [2026-10-21 21:50:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1191 [2026-10-21 21:50:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8464 [2026-10-21 21:50:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1015 [2026-10-21 21:51:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7473 [2026-10-21 21:51:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7593 [2026-10-21 21:51:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=3.6092 [2026-10-21 21:51:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4122 [2026-10-21 21:51:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9745 [2026-10-21 21:51:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2969 [2026-10-21 21:51:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6757 [2026-10-21 21:51:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0414 [2026-10-21 21:51:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0130 [2026-10-21 21:51:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7805 [2026-10-21 21:51:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4674 [2026-10-21 21:51:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9815 [2026-10-21 21:51:49][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,605 | MFU: 1.8% [2026-10-21 21:51:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1788 [2026-10-21 21:51:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4277 [2026-10-21 21:51:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3099 [2026-10-21 21:51:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3071 [2026-10-21 21:51:54][INFO][training.pretrain_engine] [Ph2|Step 378] loss=4.9905 | ppl=147.01 | aux=0.02508 | lr=2.98e-04 | gnorm=0.766 | tok/s=29,352 | MFU=7.0% | step=142898ms | tokens=1.199B | mem=4.2/32.7GB | cost=$30.98/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 08:28 (130.6h) | Phase3: SKIPPED | SFT: Oct 27 13:12 (135.3h) | DPO (Final): Oct 27 13:46 (135.9h) [2026-10-21 21:51:57][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step378 | Tokens: 1.199B | Spent: $30.98 | Remaining: $969.02 [2026-10-21 21:51:57][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step378 [2026-10-21 21:51:57][INFO][training.pretrain_engine] Starting global step 379 (accumulating 32 micro-batches)... [2026-10-21 21:51:57][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 21:51:57][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 21:51:57][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 21:51:57][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 21:51:57][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step378 to Phase-Technologies/forge-500m-pretrain (path: phase2_step378)... [2026-10-21 21:51:58][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 21:52:07][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 21:52:07][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step378 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 21:52:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0046 [2026-10-21 21:52:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1132 [2026-10-21 21:52:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3385 [2026-10-21 21:52:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5387 [2026-10-21 21:52:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1986 [2026-10-21 21:52:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0930 [2026-10-21 21:52:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9250 [2026-10-21 21:52:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9285 [2026-10-21 21:52:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5199 [2026-10-21 21:52:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4824 [2026-10-21 21:52:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4077 [2026-10-21 21:52:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=3.9798 [2026-10-21 21:53:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7870 [2026-10-21 21:53:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9501 [2026-10-21 21:53:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2116 [2026-10-21 21:53:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6841 [2026-10-21 21:53:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9353 [2026-10-21 21:53:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8392 [2026-10-21 21:53:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1689 [2026-10-21 21:53:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8637 [2026-10-21 21:53:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7123 [2026-10-21 21:53:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5946 [2026-10-21 21:53:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6582 [2026-10-21 21:53:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1093 [2026-10-21 21:53:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2122 [2026-10-21 21:53:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5284 [2026-10-21 21:53:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7461 [2026-10-21 21:54:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7340 [2026-10-21 21:54:15][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,601 | MFU: 1.8% [2026-10-21 21:54:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5853 [2026-10-21 21:54:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8692 [2026-10-21 21:54:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1971 [2026-10-21 21:54:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7552 [2026-10-21 21:54:20][INFO][training.pretrain_engine] [Ph2|Step 379] loss=4.9981 | ppl=148.13 | aux=0.02507 | lr=2.98e-04 | gnorm=0.684 | tok/s=29,353 | MFU=7.0% | step=142894ms | tokens=1.200B | mem=4.2/32.7GB | cost=$31.07/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 08:30 (130.6h) | Phase3: SKIPPED | SFT: Oct 27 13:14 (135.3h) | DPO (Final): Oct 27 13:48 (135.9h) [2026-10-21 21:54:20][INFO][training.pretrain_engine] Starting global step 380 (accumulating 32 micro-batches)... [2026-10-21 21:54:20][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 21:54:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8910 [2026-10-21 21:54:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0138 [2026-10-21 21:54:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1723 [2026-10-21 21:54:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8165 [2026-10-21 21:54:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7660 [2026-10-21 21:54:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5301 [2026-10-21 21:54:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3072 [2026-10-21 21:54:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2213 [2026-10-21 21:55:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6091 [2026-10-21 21:55:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1711 [2026-10-21 21:55:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0480 [2026-10-21 21:55:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1261 [2026-10-21 21:55:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8359 [2026-10-21 21:55:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0091 [2026-10-21 21:55:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3707 [2026-10-21 21:55:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8115 [2026-10-21 21:55:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0859 [2026-10-21 21:55:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2332 [2026-10-21 21:55:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3079 [2026-10-21 21:55:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9839 [2026-10-21 21:56:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7774 [2026-10-21 21:56:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5121 [2026-10-21 21:56:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4977 [2026-10-21 21:56:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1848 [2026-10-21 21:56:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4368 [2026-10-21 21:56:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7505 [2026-10-21 21:56:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3956 [2026-10-21 21:56:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1418 [2026-10-21 21:56:37][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,618 | MFU: 1.8% [2026-10-21 21:56:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3627 [2026-10-21 21:56:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2331 [2026-10-21 21:56:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2350 [2026-10-21 21:56:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5942 [2026-10-21 21:56:42][INFO][training.pretrain_engine] [Ph2|Step 380] loss=4.9562 | ppl=142.05 | aux=0.02507 | lr=2.98e-04 | gnorm=0.797 | tok/s=29,352 | MFU=7.0% | step=142898ms | tokens=1.201B | mem=4.2/32.7GB | cost=$31.15/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 08:32 (130.6h) | Phase3: SKIPPED | SFT: Oct 27 13:16 (135.3h) | DPO (Final): Oct 27 13:50 (135.9h) [2026-10-21 21:56:42][INFO][training.pretrain_engine] Starting global step 381 (accumulating 32 micro-batches)... [2026-10-21 21:56:42][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 21:57:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3043 [2026-10-21 21:57:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7755 [2026-10-21 21:57:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7179 [2026-10-21 21:57:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3021 [2026-10-21 21:57:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6677 [2026-10-21 21:57:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0200 [2026-10-21 21:57:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0273 [2026-10-21 21:57:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2909 [2026-10-21 21:57:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3079 [2026-10-21 21:57:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8529 [2026-10-21 21:57:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8599 [2026-10-21 21:57:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9503 [2026-10-21 21:57:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.0761 [2026-10-21 21:57:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5705 [2026-10-21 21:57:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8149 [2026-10-21 21:57:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0428 [2026-10-21 21:58:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2136 [2026-10-21 21:58:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8867 [2026-10-21 21:58:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0202 [2026-10-21 21:58:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3663 [2026-10-21 21:58:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0749 [2026-10-21 21:58:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9977 [2026-10-21 21:58:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1494 [2026-10-21 21:58:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4338 [2026-10-21 21:58:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9696 [2026-10-21 21:58:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0763 [2026-10-21 21:58:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7177 [2026-10-21 21:58:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2348 [2026-10-21 21:59:00][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,607 | MFU: 1.8% [2026-10-21 21:59:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7646 [2026-10-21 21:59:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5127 [2026-10-21 21:59:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0969 [2026-10-21 21:59:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2504 [2026-10-21 21:59:05][INFO][training.pretrain_engine] [Ph2|Step 381] loss=4.9099 | ppl=135.62 | aux=0.02506 | lr=2.98e-04 | gnorm=0.566 | tok/s=29,351 | MFU=7.0% | step=142903ms | tokens=1.202B | mem=4.2/32.7GB | cost=$31.23/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 08:34 (130.6h) | Phase3: SKIPPED | SFT: Oct 27 13:18 (135.3h) | DPO (Final): Oct 27 13:52 (135.9h) [2026-10-21 21:59:05][INFO][training.pretrain_engine] Starting global step 382 (accumulating 32 micro-batches)... [2026-10-21 21:59:05][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 21:59:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4009 [2026-10-21 21:59:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8344 [2026-10-21 21:59:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0075 [2026-10-21 21:59:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4584 [2026-10-21 21:59:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3005 [2026-10-21 21:59:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7829 [2026-10-21 21:59:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0502 [2026-10-21 21:59:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1000 [2026-10-21 21:59:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9566 [2026-10-21 21:59:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1846 [2026-10-21 21:59:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2265 [2026-10-21 21:59:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9705 [2026-10-21 22:00:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6311 [2026-10-21 22:00:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5736 [2026-10-21 22:00:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0137 [2026-10-21 22:00:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4082 [2026-10-21 22:00:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6076 [2026-10-21 22:00:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9676 [2026-10-21 22:00:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8010 [2026-10-21 22:00:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8914 [2026-10-21 22:00:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1604 [2026-10-21 22:00:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5141 [2026-10-21 22:00:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3489 [2026-10-21 22:00:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9100 [2026-10-21 22:01:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2630 [2026-10-21 22:01:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0305 [2026-10-21 22:01:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2252 [2026-10-21 22:01:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1843 [2026-10-21 22:01:24][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,585 | MFU: 1.8% [2026-10-21 22:01:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0465 [2026-10-21 22:01:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2851 [2026-10-21 22:01:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1276 [2026-10-21 22:01:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2283 [2026-10-21 22:01:29][INFO][training.pretrain_engine] [Ph2|Step 382] loss=5.0941 | ppl=163.06 | aux=0.02506 | lr=2.98e-04 | gnorm=0.645 | tok/s=29,348 | MFU=7.0% | step=142915ms | tokens=1.203B | mem=4.2/32.7GB | cost=$31.31/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 08:36 (130.6h) | Phase3: SKIPPED | SFT: Oct 27 13:20 (135.3h) | DPO (Final): Oct 27 13:54 (135.9h) [2026-10-21 22:01:29][INFO][training.pretrain_engine] Starting global step 383 (accumulating 32 micro-batches)... [2026-10-21 22:01:29][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 22:01:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7228 [2026-10-21 22:01:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7350 [2026-10-21 22:01:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3051 [2026-10-21 22:01:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0066 [2026-10-21 22:02:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2691 [2026-10-21 22:02:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8725 [2026-10-21 22:02:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7548 [2026-10-21 22:02:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1664 [2026-10-21 22:02:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9021 [2026-10-21 22:02:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3629 [2026-10-21 22:02:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1556 [2026-10-21 22:02:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1311 [2026-10-21 22:02:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1479 [2026-10-21 22:02:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0515 [2026-10-21 22:02:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0148 [2026-10-21 22:02:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9961 [2026-10-21 22:02:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0315 [2026-10-21 22:02:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8373 [2026-10-21 22:02:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7832 [2026-10-21 22:02:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0069 [2026-10-21 22:03:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2258 [2026-10-21 22:03:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4791 [2026-10-21 22:03:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8111 [2026-10-21 22:03:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3392 [2026-10-21 22:03:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2498 [2026-10-21 22:03:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9368 [2026-10-21 22:03:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7967 [2026-10-21 22:03:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9500 [2026-10-21 22:03:48][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,582 | MFU: 1.8% [2026-10-21 22:03:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0713 [2026-10-21 22:03:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4549 [2026-10-21 22:03:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5079 [2026-10-21 22:03:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2292 [2026-10-21 22:03:53][INFO][training.pretrain_engine] [Ph2|Step 383] loss=5.1368 | ppl=170.18 | aux=0.02506 | lr=2.98e-04 | gnorm=0.465 | tok/s=29,346 | MFU=7.0% | step=142925ms | tokens=1.204B | mem=4.2/32.7GB | cost=$31.39/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 08:39 (130.6h) | Phase3: SKIPPED | SFT: Oct 27 13:23 (135.3h) | DPO (Final): Oct 27 13:57 (135.9h) [2026-10-21 22:03:53][INFO][training.pretrain_engine] Starting global step 384 (accumulating 32 micro-batches)... [2026-10-21 22:03:53][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 22:04:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4557 [2026-10-21 22:04:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3554 [2026-10-21 22:04:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6012 [2026-10-21 22:04:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1535 [2026-10-21 22:04:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6703 [2026-10-21 22:04:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3087 [2026-10-21 22:04:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8867 [2026-10-21 22:04:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2015 [2026-10-21 22:04:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4650 [2026-10-21 22:04:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4651 [2026-10-21 22:04:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6423 [2026-10-21 22:04:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0865 [2026-10-21 22:05:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0254 [2026-10-21 22:05:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0981 [2026-10-21 22:05:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6521 [2026-10-21 22:05:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8202 [2026-10-21 22:05:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7881 [2026-10-21 22:05:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1566 [2026-10-21 22:05:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2757 [2026-10-21 22:05:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0973 [2026-10-21 22:05:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9516 [2026-10-21 22:05:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5559 [2026-10-21 22:05:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1114 [2026-10-21 22:05:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7203 [2026-10-21 22:05:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9756 [2026-10-21 22:05:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1061 [2026-10-21 22:05:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9459 [2026-10-21 22:05:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0281 [2026-10-21 22:06:11][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,593 | MFU: 1.8% [2026-10-21 22:06:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3769 [2026-10-21 22:06:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0690 [2026-10-21 22:06:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5541 [2026-10-21 22:06:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3226 [2026-10-21 22:06:16][INFO][training.pretrain_engine] [Ph2|Step 384] loss=4.9986 | ppl=148.21 | aux=0.02506 | lr=2.98e-04 | gnorm=0.602 | tok/s=29,347 | MFU=7.0% | step=142922ms | tokens=1.205B | mem=4.2/32.7GB | cost=$31.47/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 08:40 (130.6h) | Phase3: SKIPPED | SFT: Oct 27 13:24 (135.3h) | DPO (Final): Oct 27 13:58 (135.9h) [2026-10-21 22:06:16][INFO][training.pretrain_engine] Starting global step 385 (accumulating 32 micro-batches)... [2026-10-21 22:06:16][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 22:06:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9164 [2026-10-21 22:06:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1705 [2026-10-21 22:06:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7951 [2026-10-21 22:06:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.7156 [2026-10-21 22:06:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5600 [2026-10-21 22:06:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9461 [2026-10-21 22:06:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5324 [2026-10-21 22:06:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7570 [2026-10-21 22:07:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9002 [2026-10-21 22:07:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9537 [2026-10-21 22:07:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3967 [2026-10-21 22:07:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1327 [2026-10-21 22:07:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7063 [2026-10-21 22:07:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7368 [2026-10-21 22:07:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.0082 [2026-10-21 22:07:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=3.9071 [2026-10-21 22:07:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.0836 [2026-10-21 22:07:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2205 [2026-10-21 22:07:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8734 [2026-10-21 22:07:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1686 [2026-10-21 22:08:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5101 [2026-10-21 22:08:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4854 [2026-10-21 22:08:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4394 [2026-10-21 22:08:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6160 [2026-10-21 22:08:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3656 [2026-10-21 22:08:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8290 [2026-10-21 22:08:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7240 [2026-10-21 22:08:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3679 [2026-10-21 22:08:33][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,606 | MFU: 1.8% [2026-10-21 22:08:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4644 [2026-10-21 22:08:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2525 [2026-10-21 22:08:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3853 [2026-10-21 22:08:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6909 [2026-10-21 22:08:39][INFO][training.pretrain_engine] [Ph2|Step 385] loss=4.9386 | ppl=139.58 | aux=0.02506 | lr=2.98e-04 | gnorm=0.496 | tok/s=29,346 | MFU=7.0% | step=142925ms | tokens=1.206B | mem=4.2/32.7GB | cost=$31.56/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 08:42 (130.6h) | Phase3: SKIPPED | SFT: Oct 27 13:26 (135.3h) | DPO (Final): Oct 27 14:00 (135.9h) [2026-10-21 22:08:39][INFO][training.pretrain_engine] Starting global step 386 (accumulating 32 micro-batches)... [2026-10-21 22:08:39][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 22:08:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1411 [2026-10-21 22:08:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7386 [2026-10-21 22:08:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2785 [2026-10-21 22:08:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1737 [2026-10-21 22:09:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8079 [2026-10-21 22:09:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2375 [2026-10-21 22:09:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0359 [2026-10-21 22:09:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2329 [2026-10-21 22:09:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8548 [2026-10-21 22:09:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0086 [2026-10-21 22:09:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4905 [2026-10-21 22:09:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4885 [2026-10-21 22:09:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4705 [2026-10-21 22:09:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4891 [2026-10-21 22:09:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2458 [2026-10-21 22:09:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4325 [2026-10-21 22:10:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1905 [2026-10-21 22:10:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0315 [2026-10-21 22:10:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9677 [2026-10-21 22:10:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1944 [2026-10-21 22:10:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2454 [2026-10-21 22:10:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2040 [2026-10-21 22:10:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4140 [2026-10-21 22:10:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1296 [2026-10-21 22:10:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8859 [2026-10-21 22:10:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2264 [2026-10-21 22:10:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4241 [2026-10-21 22:10:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2963 [2026-10-21 22:10:56][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,616 | MFU: 1.8% [2026-10-21 22:11:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6517 [2026-10-21 22:11:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5908 [2026-10-21 22:11:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.1940 [2026-10-21 22:11:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8040 [2026-10-21 22:11:02][INFO][training.pretrain_engine] [Ph2|Step 386] loss=4.9930 | ppl=147.38 | aux=0.02506 | lr=2.98e-04 | gnorm=0.594 | tok/s=29,346 | MFU=7.0% | step=142928ms | tokens=1.207B | mem=4.2/32.7GB | cost=$31.64/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 08:44 (130.6h) | Phase3: SKIPPED | SFT: Oct 27 13:28 (135.3h) | DPO (Final): Oct 27 14:02 (135.9h) [2026-10-21 22:11:02][INFO][training.pretrain_engine] Starting global step 387 (accumulating 32 micro-batches)... [2026-10-21 22:11:02][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 22:11:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7758 [2026-10-21 22:11:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2144 [2026-10-21 22:11:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3792 [2026-10-21 22:11:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0289 [2026-10-21 22:11:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3368 [2026-10-21 22:11:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9759 [2026-10-21 22:11:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6068 [2026-10-21 22:11:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3382 [2026-10-21 22:11:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0372 [2026-10-21 22:11:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1687 [2026-10-21 22:11:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0743 [2026-10-21 22:11:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1268 [2026-10-21 22:12:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7338 [2026-10-21 22:12:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=3.9643 [2026-10-21 22:12:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7618 [2026-10-21 22:12:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7677 [2026-10-21 22:12:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9316 [2026-10-21 22:12:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4283 [2026-10-21 22:12:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9842 [2026-10-21 22:12:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1369 [2026-10-21 22:12:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4340 [2026-10-21 22:12:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8933 [2026-10-21 22:12:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7348 [2026-10-21 22:12:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5381 [2026-10-21 22:13:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8108 [2026-10-21 22:13:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7615 [2026-10-21 22:13:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.0947 [2026-10-21 22:13:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9570 [2026-10-21 22:13:20][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,590 | MFU: 1.8% [2026-10-21 22:13:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4081 [2026-10-21 22:13:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6245 [2026-10-21 22:13:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7188 [2026-10-21 22:13:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5187 [2026-10-21 22:13:24][INFO][training.pretrain_engine] [Ph2|Step 387] loss=4.9992 | ppl=148.29 | aux=0.02506 | lr=2.98e-04 | gnorm=0.527 | tok/s=29,345 | MFU=7.0% | step=142931ms | tokens=1.208B | mem=4.2/32.7GB | cost=$31.72/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 08:46 (130.6h) | Phase3: SKIPPED | SFT: Oct 27 13:30 (135.3h) | DPO (Final): Oct 27 14:04 (135.9h) [2026-10-21 22:13:27][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step387 | Tokens: 1.208B | Spent: $31.72 | Remaining: $968.28 [2026-10-21 22:13:27][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step387 [2026-10-21 22:13:27][INFO][training.pretrain_engine] Starting global step 388 (accumulating 32 micro-batches)... [2026-10-21 22:13:27][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 22:13:28][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 22:13:28][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 22:13:28][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 22:13:28][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step387 to Phase-Technologies/forge-500m-pretrain (path: phase2_step387)... [2026-10-21 22:13:28][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 22:13:37][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 22:13:37][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step387 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 22:13:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2379 [2026-10-21 22:13:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2178 [2026-10-21 22:13:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9542 [2026-10-21 22:13:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7849 [2026-10-21 22:14:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9833 [2026-10-21 22:14:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9481 [2026-10-21 22:14:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7154 [2026-10-21 22:14:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9498 [2026-10-21 22:14:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5240 [2026-10-21 22:14:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3444 [2026-10-21 22:14:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2897 [2026-10-21 22:14:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7663 [2026-10-21 22:14:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1227 [2026-10-21 22:14:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5581 [2026-10-21 22:14:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9840 [2026-10-21 22:14:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0154 [2026-10-21 22:14:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9108 [2026-10-21 22:14:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1024 [2026-10-21 22:14:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2426 [2026-10-21 22:14:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7732 [2026-10-21 22:15:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0188 [2026-10-21 22:15:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7173 [2026-10-21 22:15:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9668 [2026-10-21 22:15:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1181 [2026-10-21 22:15:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0720 [2026-10-21 22:15:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1590 [2026-10-21 22:15:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5491 [2026-10-21 22:15:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1072 [2026-10-21 22:15:45][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,603 | MFU: 1.8% [2026-10-21 22:15:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0737 [2026-10-21 22:15:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0545 [2026-10-21 22:15:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7642 [2026-10-21 22:15:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4900 [2026-10-21 22:15:50][INFO][training.pretrain_engine] [Ph2|Step 388] loss=5.0751 | ppl=159.98 | aux=0.02506 | lr=2.98e-04 | gnorm=0.488 | tok/s=29,345 | MFU=7.0% | step=142931ms | tokens=1.209B | mem=4.2/32.7GB | cost=$31.80/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 08:48 (130.5h) | Phase3: SKIPPED | SFT: Oct 27 13:32 (135.3h) | DPO (Final): Oct 27 14:06 (135.8h) [2026-10-21 22:15:50][INFO][training.pretrain_engine] Starting global step 389 (accumulating 32 micro-batches)... [2026-10-21 22:15:50][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 22:16:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7934 [2026-10-21 22:16:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0273 [2026-10-21 22:16:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8426 [2026-10-21 22:16:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7639 [2026-10-21 22:16:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1378 [2026-10-21 22:16:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9541 [2026-10-21 22:16:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0592 [2026-10-21 22:16:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0465 [2026-10-21 22:16:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0593 [2026-10-21 22:16:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0180 [2026-10-21 22:16:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2981 [2026-10-21 22:16:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0616 [2026-10-21 22:17:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0722 [2026-10-21 22:17:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8189 [2026-10-21 22:17:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0062 [2026-10-21 22:17:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6558 [2026-10-21 22:17:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8384 [2026-10-21 22:17:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1278 [2026-10-21 22:17:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9670 [2026-10-21 22:17:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5955 [2026-10-21 22:17:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3167 [2026-10-21 22:17:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6871 [2026-10-21 22:17:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4907 [2026-10-21 22:17:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6149 [2026-10-21 22:17:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8806 [2026-10-21 22:17:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2850 [2026-10-21 22:17:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9939 [2026-10-21 22:17:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9499 [2026-10-21 22:18:08][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,609 | MFU: 1.8% [2026-10-21 22:18:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9156 [2026-10-21 22:18:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8711 [2026-10-21 22:18:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8278 [2026-10-21 22:18:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0860 [2026-10-21 22:18:13][INFO][training.pretrain_engine] [Ph2|Step 389] loss=5.0070 | ppl=149.45 | aux=0.02506 | lr=2.98e-04 | gnorm=0.471 | tok/s=29,344 | MFU=7.0% | step=142938ms | tokens=1.210B | mem=4.2/32.7GB | cost=$31.89/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 08:50 (130.5h) | Phase3: SKIPPED | SFT: Oct 27 13:34 (135.3h) | DPO (Final): Oct 27 14:08 (135.8h) [2026-10-21 22:18:13][INFO][training.pretrain_engine] Starting global step 390 (accumulating 32 micro-batches)... [2026-10-21 22:18:13][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 22:18:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3196 [2026-10-21 22:18:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0321 [2026-10-21 22:18:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2888 [2026-10-21 22:18:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9870 [2026-10-21 22:18:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2941 [2026-10-21 22:18:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1377 [2026-10-21 22:18:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7095 [2026-10-21 22:18:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4255 [2026-10-21 22:19:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0715 [2026-10-21 22:19:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9433 [2026-10-21 22:19:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0781 [2026-10-21 22:19:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9112 [2026-10-21 22:19:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1912 [2026-10-21 22:19:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3771 [2026-10-21 22:19:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0623 [2026-10-21 22:19:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3233 [2026-10-21 22:19:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2730 [2026-10-21 22:19:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2428 [2026-10-21 22:19:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9009 [2026-10-21 22:19:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0209 [2026-10-21 22:19:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8886 [2026-10-21 22:19:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2598 [2026-10-21 22:20:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8537 [2026-10-21 22:20:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7714 [2026-10-21 22:20:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1051 [2026-10-21 22:20:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6610 [2026-10-21 22:20:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.1791 [2026-10-21 22:20:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3632 [2026-10-21 22:20:30][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,625 | MFU: 1.8% [2026-10-21 22:20:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0288 [2026-10-21 22:20:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8293 [2026-10-21 22:20:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7274 [2026-10-21 22:20:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7037 [2026-10-21 22:20:35][INFO][training.pretrain_engine] [Ph2|Step 390] loss=5.0637 | ppl=158.17 | aux=0.02507 | lr=2.98e-04 | gnorm=0.504 | tok/s=29,344 | MFU=7.0% | step=142937ms | tokens=1.211B | mem=4.2/32.7GB | cost=$31.97/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 08:52 (130.5h) | Phase3: SKIPPED | SFT: Oct 27 13:36 (135.3h) | DPO (Final): Oct 27 14:10 (135.8h) [2026-10-21 22:20:35][INFO][training.pretrain_engine] Starting global step 391 (accumulating 32 micro-batches)... [2026-10-21 22:20:35][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 22:20:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9564 [2026-10-21 22:20:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2015 [2026-10-21 22:20:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0010 [2026-10-21 22:20:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6177 [2026-10-21 22:21:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5850 [2026-10-21 22:21:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8182 [2026-10-21 22:21:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2272 [2026-10-21 22:21:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6126 [2026-10-21 22:21:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1682 [2026-10-21 22:21:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1606 [2026-10-21 22:21:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7422 [2026-10-21 22:21:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4481 [2026-10-21 22:21:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0646 [2026-10-21 22:21:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6588 [2026-10-21 22:21:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0914 [2026-10-21 22:21:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6504 [2026-10-21 22:22:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0665 [2026-10-21 22:22:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0091 [2026-10-21 22:22:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3021 [2026-10-21 22:22:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0951 [2026-10-21 22:22:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0627 [2026-10-21 22:22:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9451 [2026-10-21 22:22:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0283 [2026-10-21 22:22:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7404 [2026-10-21 22:22:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0878 [2026-10-21 22:22:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0030 [2026-10-21 22:22:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2821 [2026-10-21 22:22:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2496 [2026-10-21 22:22:53][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,609 | MFU: 1.8% [2026-10-21 22:22:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5658 [2026-10-21 22:22:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0587 [2026-10-21 22:22:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0289 [2026-10-21 22:22:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4936 [2026-10-21 22:22:59][INFO][training.pretrain_engine] [Ph2|Step 391] loss=5.0696 | ppl=159.10 | aux=0.02505 | lr=2.98e-04 | gnorm=0.504 | tok/s=29,343 | MFU=7.0% | step=142941ms | tokens=1.212B | mem=4.2/32.7GB | cost=$32.05/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 08:54 (130.5h) | Phase3: SKIPPED | SFT: Oct 27 13:38 (135.3h) | DPO (Final): Oct 27 14:12 (135.8h) [2026-10-21 22:22:59][INFO][training.pretrain_engine] Starting global step 392 (accumulating 32 micro-batches)... [2026-10-21 22:22:59][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 22:23:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1510 [2026-10-21 22:23:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2369 [2026-10-21 22:23:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0519 [2026-10-21 22:23:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3641 [2026-10-21 22:23:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8583 [2026-10-21 22:23:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7191 [2026-10-21 22:23:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1585 [2026-10-21 22:23:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2714 [2026-10-21 22:23:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3715 [2026-10-21 22:23:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0657 [2026-10-21 22:23:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4197 [2026-10-21 22:23:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2238 [2026-10-21 22:24:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2277 [2026-10-21 22:24:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3206 [2026-10-21 22:24:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2292 [2026-10-21 22:24:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3157 [2026-10-21 22:24:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3027 [2026-10-21 22:24:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0356 [2026-10-21 22:24:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1608 [2026-10-21 22:24:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4898 [2026-10-21 22:24:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6131 [2026-10-21 22:24:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4606 [2026-10-21 22:24:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3006 [2026-10-21 22:24:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5009 [2026-10-21 22:25:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9821 [2026-10-21 22:25:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2196 [2026-10-21 22:25:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5359 [2026-10-21 22:25:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0578 [2026-10-21 22:25:16][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,603 | MFU: 1.8% [2026-10-21 22:25:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0503 [2026-10-21 22:25:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2603 [2026-10-21 22:25:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3106 [2026-10-21 22:25:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9668 [2026-10-21 22:25:22][INFO][training.pretrain_engine] [Ph2|Step 392] loss=5.0780 | ppl=160.44 | aux=0.02506 | lr=2.98e-04 | gnorm=0.535 | tok/s=29,341 | MFU=7.0% | step=142951ms | tokens=1.213B | mem=4.2/32.7GB | cost=$32.13/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 08:56 (130.5h) | Phase3: SKIPPED | SFT: Oct 27 13:40 (135.3h) | DPO (Final): Oct 27 14:14 (135.8h) [2026-10-21 22:25:22][INFO][training.pretrain_engine] Starting global step 393 (accumulating 32 micro-batches)... [2026-10-21 22:25:22][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 22:25:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5978 [2026-10-21 22:25:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9977 [2026-10-21 22:25:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5122 [2026-10-21 22:25:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3169 [2026-10-21 22:25:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.9138 [2026-10-21 22:25:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7397 [2026-10-21 22:25:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6537 [2026-10-21 22:25:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2130 [2026-10-21 22:26:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2247 [2026-10-21 22:26:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5242 [2026-10-21 22:26:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1417 [2026-10-21 22:26:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7811 [2026-10-21 22:26:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7277 [2026-10-21 22:26:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3022 [2026-10-21 22:26:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5529 [2026-10-21 22:26:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7361 [2026-10-21 22:26:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9226 [2026-10-21 22:26:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8796 [2026-10-21 22:26:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8546 [2026-10-21 22:26:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8788 [2026-10-21 22:27:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2832 [2026-10-21 22:27:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3380 [2026-10-21 22:27:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1771 [2026-10-21 22:27:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7575 [2026-10-21 22:27:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7558 [2026-10-21 22:27:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8976 [2026-10-21 22:27:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9772 [2026-10-21 22:27:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0353 [2026-10-21 22:27:40][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,608 | MFU: 1.8% [2026-10-21 22:27:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7675 [2026-10-21 22:27:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3629 [2026-10-21 22:27:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9974 [2026-10-21 22:27:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2098 [2026-10-21 22:27:45][INFO][training.pretrain_engine] [Ph2|Step 393] loss=4.9284 | ppl=138.15 | aux=0.02507 | lr=2.98e-04 | gnorm=0.508 | tok/s=29,339 | MFU=7.0% | step=142958ms | tokens=1.214B | mem=4.2/32.7GB | cost=$32.21/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 08:58 (130.5h) | Phase3: SKIPPED | SFT: Oct 27 13:42 (135.3h) | DPO (Final): Oct 27 14:17 (135.8h) [2026-10-21 22:27:45][INFO][training.pretrain_engine] Starting global step 394 (accumulating 32 micro-batches)... [2026-10-21 22:27:45][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 22:28:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3924 [2026-10-21 22:28:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5656 [2026-10-21 22:28:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2891 [2026-10-21 22:28:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4857 [2026-10-21 22:28:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2449 [2026-10-21 22:28:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9122 [2026-10-21 22:28:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9630 [2026-10-21 22:28:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1053 [2026-10-21 22:28:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2119 [2026-10-21 22:28:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1612 [2026-10-21 22:28:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9851 [2026-10-21 22:28:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2504 [2026-10-21 22:28:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5973 [2026-10-21 22:28:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0794 [2026-10-21 22:28:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2737 [2026-10-21 22:28:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2350 [2026-10-21 22:29:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9429 [2026-10-21 22:29:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6820 [2026-10-21 22:29:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7572 [2026-10-21 22:29:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0212 [2026-10-21 22:29:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0276 [2026-10-21 22:29:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7465 [2026-10-21 22:29:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8658 [2026-10-21 22:29:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0705 [2026-10-21 22:29:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1408 [2026-10-21 22:29:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2332 [2026-10-21 22:29:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0933 [2026-10-21 22:29:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0289 [2026-10-21 22:30:03][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,594 | MFU: 1.8% [2026-10-21 22:30:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.1606 [2026-10-21 22:30:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9603 [2026-10-21 22:30:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2777 [2026-10-21 22:30:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1181 [2026-10-21 22:30:08][INFO][training.pretrain_engine] [Ph2|Step 394] loss=5.1141 | ppl=166.35 | aux=0.02505 | lr=2.98e-04 | gnorm=0.516 | tok/s=29,337 | MFU=7.0% | step=142970ms | tokens=1.215B | mem=4.2/32.7GB | cost=$32.29/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 09:01 (130.5h) | Phase3: SKIPPED | SFT: Oct 27 13:45 (135.3h) | DPO (Final): Oct 27 14:19 (135.8h) [2026-10-21 22:30:08][INFO][training.pretrain_engine] Starting global step 395 (accumulating 32 micro-batches)... [2026-10-21 22:30:08][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 22:30:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8232 [2026-10-21 22:30:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9000 [2026-10-21 22:30:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4928 [2026-10-21 22:30:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1259 [2026-10-21 22:30:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5311 [2026-10-21 22:30:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3023 [2026-10-21 22:30:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7354 [2026-10-21 22:30:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1230 [2026-10-21 22:31:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1307 [2026-10-21 22:31:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2739 [2026-10-21 22:31:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9210 [2026-10-21 22:31:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2519 [2026-10-21 22:31:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0049 [2026-10-21 22:31:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0033 [2026-10-21 22:31:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5853 [2026-10-21 22:31:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4594 [2026-10-21 22:31:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0920 [2026-10-21 22:31:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8005 [2026-10-21 22:31:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6087 [2026-10-21 22:31:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9104 [2026-10-21 22:31:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8446 [2026-10-21 22:31:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1683 [2026-10-21 22:31:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6989 [2026-10-21 22:31:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8679 [2026-10-21 22:32:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2508 [2026-10-21 22:32:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5817 [2026-10-21 22:32:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0356 [2026-10-21 22:32:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2588 [2026-10-21 22:32:26][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,617 | MFU: 1.8% [2026-10-21 22:32:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1096 [2026-10-21 22:32:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8931 [2026-10-21 22:32:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0902 [2026-10-21 22:32:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7792 [2026-10-21 22:32:31][INFO][training.pretrain_engine] [Ph2|Step 395] loss=5.0899 | ppl=162.37 | aux=0.02505 | lr=2.98e-04 | gnorm=0.559 | tok/s=29,337 | MFU=7.0% | step=142968ms | tokens=1.216B | mem=4.2/32.7GB | cost=$32.38/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 09:03 (130.5h) | Phase3: SKIPPED | SFT: Oct 27 13:47 (135.2h) | DPO (Final): Oct 27 14:21 (135.8h) [2026-10-21 22:32:31][INFO][training.pretrain_engine] Starting global step 396 (accumulating 32 micro-batches)... [2026-10-21 22:32:31][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 22:32:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2900 [2026-10-21 22:32:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2374 [2026-10-21 22:32:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0367 [2026-10-21 22:32:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4744 [2026-10-21 22:33:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1327 [2026-10-21 22:33:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9538 [2026-10-21 22:33:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6800 [2026-10-21 22:33:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1697 [2026-10-21 22:33:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6421 [2026-10-21 22:33:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1369 [2026-10-21 22:33:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5459 [2026-10-21 22:33:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3817 [2026-10-21 22:33:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5032 [2026-10-21 22:33:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3056 [2026-10-21 22:33:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7995 [2026-10-21 22:33:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3822 [2026-10-21 22:33:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2750 [2026-10-21 22:34:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.2794 [2026-10-21 22:34:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8998 [2026-10-21 22:34:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6839 [2026-10-21 22:34:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2545 [2026-10-21 22:34:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2227 [2026-10-21 22:34:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2759 [2026-10-21 22:34:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9587 [2026-10-21 22:34:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4068 [2026-10-21 22:34:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9281 [2026-10-21 22:34:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4607 [2026-10-21 22:34:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2776 [2026-10-21 22:34:48][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,620 | MFU: 1.8% [2026-10-21 22:34:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8357 [2026-10-21 22:34:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1074 [2026-10-21 22:34:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7581 [2026-10-21 22:34:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4436 [2026-10-21 22:34:54][INFO][training.pretrain_engine] [Ph2|Step 396] loss=5.0844 | ppl=161.48 | aux=0.02506 | lr=2.98e-04 | gnorm=0.676 | tok/s=29,337 | MFU=7.0% | step=142971ms | tokens=1.217B | mem=4.2/32.7GB | cost=$32.46/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 09:05 (130.5h) | Phase3: SKIPPED | SFT: Oct 27 13:49 (135.2h) | DPO (Final): Oct 27 14:23 (135.8h) [2026-10-21 22:34:57][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step396 | Tokens: 1.217B | Spent: $32.46 | Remaining: $967.54 [2026-10-21 22:34:57][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step396 [2026-10-21 22:34:57][INFO][training.pretrain_engine] Starting global step 397 (accumulating 32 micro-batches)... [2026-10-21 22:34:57][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 22:34:57][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 22:34:57][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 22:34:57][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 22:34:57][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step396 to Phase-Technologies/forge-500m-pretrain (path: phase2_step396)... [2026-10-21 22:34:57][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 22:35:07][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 22:35:07][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step396 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 22:35:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6901 [2026-10-21 22:35:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0009 [2026-10-21 22:35:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7882 [2026-10-21 22:35:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3151 [2026-10-21 22:35:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3800 [2026-10-21 22:35:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2468 [2026-10-21 22:35:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4732 [2026-10-21 22:35:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7699 [2026-10-21 22:35:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4494 [2026-10-21 22:35:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5505 [2026-10-21 22:35:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5728 [2026-10-21 22:35:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9245 [2026-10-21 22:36:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3906 [2026-10-21 22:36:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8778 [2026-10-21 22:36:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8938 [2026-10-21 22:36:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2661 [2026-10-21 22:36:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5715 [2026-10-21 22:36:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1500 [2026-10-21 22:36:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2825 [2026-10-21 22:36:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5092 [2026-10-21 22:36:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6671 [2026-10-21 22:36:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9065 [2026-10-21 22:36:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3818 [2026-10-21 22:36:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9653 [2026-10-21 22:36:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0632 [2026-10-21 22:36:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6280 [2026-10-21 22:36:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3255 [2026-10-21 22:37:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2648 [2026-10-21 22:37:15][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,612 | MFU: 1.8% [2026-10-21 22:37:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8988 [2026-10-21 22:37:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5699 [2026-10-21 22:37:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7967 [2026-10-21 22:37:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6937 [2026-10-21 22:37:19][INFO][training.pretrain_engine] [Ph2|Step 397] loss=4.9929 | ppl=147.36 | aux=0.02506 | lr=2.98e-04 | gnorm=0.734 | tok/s=29,338 | MFU=7.0% | step=142967ms | tokens=1.218B | mem=4.2/32.7GB | cost=$32.54/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 09:06 (130.5h) | Phase3: SKIPPED | SFT: Oct 27 13:50 (135.2h) | DPO (Final): Oct 27 14:24 (135.8h) [2026-10-21 22:37:19][INFO][training.pretrain_engine] Starting global step 398 (accumulating 32 micro-batches)... [2026-10-21 22:37:19][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 22:37:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7608 [2026-10-21 22:37:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8310 [2026-10-21 22:37:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2704 [2026-10-21 22:37:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7675 [2026-10-21 22:37:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2385 [2026-10-21 22:37:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0451 [2026-10-21 22:37:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2256 [2026-10-21 22:37:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0525 [2026-10-21 22:38:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3205 [2026-10-21 22:38:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5627 [2026-10-21 22:38:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3035 [2026-10-21 22:38:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4628 [2026-10-21 22:38:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3408 [2026-10-21 22:38:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5950 [2026-10-21 22:38:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5024 [2026-10-21 22:38:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3180 [2026-10-21 22:38:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5443 [2026-10-21 22:38:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7637 [2026-10-21 22:38:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7767 [2026-10-21 22:38:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0591 [2026-10-21 22:39:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0508 [2026-10-21 22:39:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3794 [2026-10-21 22:39:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4231 [2026-10-21 22:39:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0689 [2026-10-21 22:39:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2417 [2026-10-21 22:39:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6183 [2026-10-21 22:39:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3853 [2026-10-21 22:39:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3442 [2026-10-21 22:39:37][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,607 | MFU: 1.8% [2026-10-21 22:39:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0917 [2026-10-21 22:39:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6999 [2026-10-21 22:39:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8998 [2026-10-21 22:39:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4002 [2026-10-21 22:39:42][INFO][training.pretrain_engine] [Ph2|Step 398] loss=5.1362 | ppl=170.07 | aux=0.02506 | lr=2.98e-04 | gnorm=0.781 | tok/s=29,335 | MFU=7.0% | step=142977ms | tokens=1.219B | mem=4.2/32.7GB | cost=$32.62/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 09:08 (130.5h) | Phase3: SKIPPED | SFT: Oct 27 13:53 (135.2h) | DPO (Final): Oct 27 14:27 (135.8h) [2026-10-21 22:39:42][INFO][training.pretrain_engine] Starting global step 399 (accumulating 32 micro-batches)... [2026-10-21 22:39:42][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 22:40:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0451 [2026-10-21 22:40:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.0980 [2026-10-21 22:40:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0299 [2026-10-21 22:40:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9505 [2026-10-21 22:40:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5958 [2026-10-21 22:40:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5232 [2026-10-21 22:40:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2033 [2026-10-21 22:40:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2458 [2026-10-21 22:40:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4393 [2026-10-21 22:40:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1553 [2026-10-21 22:40:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7550 [2026-10-21 22:40:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3262 [2026-10-21 22:40:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4832 [2026-10-21 22:40:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.0032 [2026-10-21 22:40:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4808 [2026-10-21 22:40:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.1691 [2026-10-21 22:41:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3244 [2026-10-21 22:41:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0611 [2026-10-21 22:41:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9941 [2026-10-21 22:41:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8929 [2026-10-21 22:41:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2930 [2026-10-21 22:41:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5985 [2026-10-21 22:41:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6607 [2026-10-21 22:41:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1722 [2026-10-21 22:41:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8809 [2026-10-21 22:41:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0833 [2026-10-21 22:41:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4173 [2026-10-21 22:41:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5895 [2026-10-21 22:42:00][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,605 | MFU: 1.8% [2026-10-21 22:42:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1239 [2026-10-21 22:42:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8907 [2026-10-21 22:42:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0824 [2026-10-21 22:42:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9805 [2026-10-21 22:42:05][INFO][training.pretrain_engine] [Ph2|Step 399] loss=5.0031 | ppl=148.87 | aux=0.02506 | lr=2.98e-04 | gnorm=0.914 | tok/s=29,335 | MFU=7.0% | step=142978ms | tokens=1.221B | mem=4.2/32.7GB | cost=$32.71/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 09:10 (130.5h) | Phase3: SKIPPED | SFT: Oct 27 13:54 (135.2h) | DPO (Final): Oct 27 14:28 (135.8h) [2026-10-21 22:42:05][INFO][training.pretrain_engine] Starting global step 400 (accumulating 32 micro-batches)... [2026-10-21 22:42:05][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 22:42:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3418 [2026-10-21 22:42:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5193 [2026-10-21 22:42:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2629 [2026-10-21 22:42:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1394 [2026-10-21 22:42:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0807 [2026-10-21 22:42:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6726 [2026-10-21 22:42:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4622 [2026-10-21 22:42:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7682 [2026-10-21 22:42:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9661 [2026-10-21 22:42:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0413 [2026-10-21 22:42:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1433 [2026-10-21 22:42:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7021 [2026-10-21 22:43:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5889 [2026-10-21 22:43:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1148 [2026-10-21 22:43:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6647 [2026-10-21 22:43:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2261 [2026-10-21 22:43:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9195 [2026-10-21 22:43:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4639 [2026-10-21 22:43:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9026 [2026-10-21 22:43:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0921 [2026-10-21 22:43:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3043 [2026-10-21 22:43:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1930 [2026-10-21 22:43:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3131 [2026-10-21 22:43:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9740 [2026-10-21 22:44:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3174 [2026-10-21 22:44:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7342 [2026-10-21 22:44:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1474 [2026-10-21 22:44:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3584 [2026-10-21 22:44:24][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,586 | MFU: 1.8% [2026-10-21 22:44:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9356 [2026-10-21 22:44:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6991 [2026-10-21 22:44:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8037 [2026-10-21 22:44:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9711 [2026-10-21 22:44:28][INFO][training.pretrain_engine] [Ph2|Step 400] loss=5.0089 | ppl=149.75 | aux=0.02506 | lr=2.98e-04 | gnorm=0.934 | tok/s=29,336 | MFU=7.0% | step=142976ms | tokens=1.222B | mem=4.2/32.7GB | cost=$32.79/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 09:12 (130.5h) | Phase3: SKIPPED | SFT: Oct 27 13:56 (135.2h) | DPO (Final): Oct 27 14:30 (135.8h) [2026-10-21 22:44:28][INFO][training.pretrain_engine] Starting global step 401 (accumulating 32 micro-batches)... [2026-10-21 22:44:28][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 22:44:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9152 [2026-10-21 22:44:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4772 [2026-10-21 22:44:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6668 [2026-10-21 22:44:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7109 [2026-10-21 22:45:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8584 [2026-10-21 22:45:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1863 [2026-10-21 22:45:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1265 [2026-10-21 22:45:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1791 [2026-10-21 22:45:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4694 [2026-10-21 22:45:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1201 [2026-10-21 22:45:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=6.1557 [2026-10-21 22:45:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3470 [2026-10-21 22:45:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1978 [2026-10-21 22:45:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2113 [2026-10-21 22:45:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1603 [2026-10-21 22:45:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2732 [2026-10-21 22:45:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6787 [2026-10-21 22:45:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.2005 [2026-10-21 22:45:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0406 [2026-10-21 22:45:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6959 [2026-10-21 22:46:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2772 [2026-10-21 22:46:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6087 [2026-10-21 22:46:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9870 [2026-10-21 22:46:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2708 [2026-10-21 22:46:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2677 [2026-10-21 22:46:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6621 [2026-10-21 22:46:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5245 [2026-10-21 22:46:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4951 [2026-10-21 22:46:46][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,614 | MFU: 1.8% [2026-10-21 22:46:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0351 [2026-10-21 22:46:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6075 [2026-10-21 22:46:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5486 [2026-10-21 22:46:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9375 [2026-10-21 22:46:51][INFO][training.pretrain_engine] [Ph2|Step 401] loss=4.9706 | ppl=144.12 | aux=0.02506 | lr=2.98e-04 | gnorm=0.773 | tok/s=29,335 | MFU=7.0% | step=142981ms | tokens=1.223B | mem=4.2/32.7GB | cost=$32.87/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 09:14 (130.5h) | Phase3: SKIPPED | SFT: Oct 27 13:58 (135.2h) | DPO (Final): Oct 27 14:32 (135.8h) [2026-10-21 22:46:51][INFO][training.pretrain_engine] Starting global step 402 (accumulating 32 micro-batches)... [2026-10-21 22:46:51][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 22:47:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0393 [2026-10-21 22:47:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9884 [2026-10-21 22:47:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8819 [2026-10-21 22:47:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9447 [2026-10-21 22:47:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8299 [2026-10-21 22:47:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7882 [2026-10-21 22:47:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7756 [2026-10-21 22:47:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8158 [2026-10-21 22:47:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1742 [2026-10-21 22:47:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3289 [2026-10-21 22:47:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8614 [2026-10-21 22:47:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6683 [2026-10-21 22:48:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9363 [2026-10-21 22:48:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.0611 [2026-10-21 22:48:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8108 [2026-10-21 22:48:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8703 [2026-10-21 22:48:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8811 [2026-10-21 22:48:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6790 [2026-10-21 22:48:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7261 [2026-10-21 22:48:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0922 [2026-10-21 22:48:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2571 [2026-10-21 22:48:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0295 [2026-10-21 22:48:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3077 [2026-10-21 22:48:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1282 [2026-10-21 22:48:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4880 [2026-10-21 22:48:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2200 [2026-10-21 22:48:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6723 [2026-10-21 22:48:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1964 [2026-10-21 22:49:09][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,606 | MFU: 1.8% [2026-10-21 22:49:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1130 [2026-10-21 22:49:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9084 [2026-10-21 22:49:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4399 [2026-10-21 22:49:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2254 [2026-10-21 22:49:15][INFO][training.pretrain_engine] [Ph2|Step 402] loss=4.9894 | ppl=146.84 | aux=0.02506 | lr=2.98e-04 | gnorm=0.848 | tok/s=29,333 | MFU=7.0% | step=142988ms | tokens=1.224B | mem=4.2/32.7GB | cost=$32.95/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 09:16 (130.5h) | Phase3: SKIPPED | SFT: Oct 27 14:00 (135.2h) | DPO (Final): Oct 27 14:34 (135.8h) [2026-10-21 22:49:15][INFO][training.pretrain_engine] Starting global step 403 (accumulating 32 micro-batches)... [2026-10-21 22:49:15][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 22:49:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2013 [2026-10-21 22:49:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7881 [2026-10-21 22:49:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0239 [2026-10-21 22:49:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2915 [2026-10-21 22:49:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2994 [2026-10-21 22:49:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6474 [2026-10-21 22:49:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.0450 [2026-10-21 22:49:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5725 [2026-10-21 22:50:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4801 [2026-10-21 22:50:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1382 [2026-10-21 22:50:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1996 [2026-10-21 22:50:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0926 [2026-10-21 22:50:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1458 [2026-10-21 22:50:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9528 [2026-10-21 22:50:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2281 [2026-10-21 22:50:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1038 [2026-10-21 22:50:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.2191 [2026-10-21 22:50:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.0843 [2026-10-21 22:50:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7332 [2026-10-21 22:50:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=3.2773 [2026-10-21 22:51:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7868 [2026-10-21 22:51:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3559 [2026-10-21 22:51:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3955 [2026-10-21 22:51:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1025 [2026-10-21 22:51:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8304 [2026-10-21 22:51:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2963 [2026-10-21 22:51:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5536 [2026-10-21 22:51:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0216 [2026-10-21 22:51:33][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,613 | MFU: 1.8% [2026-10-21 22:51:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0579 [2026-10-21 22:51:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7849 [2026-10-21 22:51:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0000 [2026-10-21 22:51:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0185 [2026-10-21 22:51:38][INFO][training.pretrain_engine] [Ph2|Step 403] loss=5.0376 | ppl=154.10 | aux=0.02505 | lr=2.98e-04 | gnorm=0.746 | tok/s=29,333 | MFU=7.0% | step=142987ms | tokens=1.225B | mem=4.2/32.7GB | cost=$33.03/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 09:18 (130.4h) | Phase3: SKIPPED | SFT: Oct 27 14:02 (135.2h) | DPO (Final): Oct 27 14:36 (135.8h) [2026-10-21 22:51:38][INFO][training.pretrain_engine] Starting global step 404 (accumulating 32 micro-batches)... [2026-10-21 22:51:38][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 22:51:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0355 [2026-10-21 22:51:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1111 [2026-10-21 22:51:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9958 [2026-10-21 22:51:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9569 [2026-10-21 22:52:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5217 [2026-10-21 22:52:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2115 [2026-10-21 22:52:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7618 [2026-10-21 22:52:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2167 [2026-10-21 22:52:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9483 [2026-10-21 22:52:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6897 [2026-10-21 22:52:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0092 [2026-10-21 22:52:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6962 [2026-10-21 22:52:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7996 [2026-10-21 22:52:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=3.6253 [2026-10-21 22:52:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2412 [2026-10-21 22:52:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6584 [2026-10-21 22:53:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1849 [2026-10-21 22:53:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4405 [2026-10-21 22:53:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1225 [2026-10-21 22:53:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5533 [2026-10-21 22:53:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4744 [2026-10-21 22:53:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4687 [2026-10-21 22:53:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4838 [2026-10-21 22:53:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4568 [2026-10-21 22:53:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0466 [2026-10-21 22:53:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2476 [2026-10-21 22:53:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3562 [2026-10-21 22:53:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5414 [2026-10-21 22:53:56][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,601 | MFU: 1.8% [2026-10-21 22:54:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5167 [2026-10-21 22:54:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6812 [2026-10-21 22:54:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1036 [2026-10-21 22:54:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7588 [2026-10-21 22:54:00][INFO][training.pretrain_engine] [Ph2|Step 404] loss=4.9405 | ppl=139.84 | aux=0.02507 | lr=2.98e-04 | gnorm=0.715 | tok/s=29,335 | MFU=7.0% | step=142980ms | tokens=1.226B | mem=4.2/32.7GB | cost=$33.11/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 09:19 (130.4h) | Phase3: SKIPPED | SFT: Oct 27 14:03 (135.2h) | DPO (Final): Oct 27 14:38 (135.7h) [2026-10-21 22:54:00][INFO][training.pretrain_engine] Starting global step 405 (accumulating 32 micro-batches)... [2026-10-21 22:54:00][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 22:54:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6653 [2026-10-21 22:54:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5572 [2026-10-21 22:54:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5908 [2026-10-21 22:54:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8787 [2026-10-21 22:54:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3489 [2026-10-21 22:54:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6715 [2026-10-21 22:54:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2142 [2026-10-21 22:54:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6011 [2026-10-21 22:54:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8310 [2026-10-21 22:54:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1977 [2026-10-21 22:54:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6562 [2026-10-21 22:54:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0280 [2026-10-21 22:55:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7524 [2026-10-21 22:55:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1360 [2026-10-21 22:55:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4926 [2026-10-21 22:55:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0478 [2026-10-21 22:55:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1259 [2026-10-21 22:55:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0099 [2026-10-21 22:55:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3799 [2026-10-21 22:55:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8019 [2026-10-21 22:55:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2240 [2026-10-21 22:55:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3294 [2026-10-21 22:55:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8827 [2026-10-21 22:55:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4571 [2026-10-21 22:56:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6858 [2026-10-21 22:56:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8796 [2026-10-21 22:56:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8195 [2026-10-21 22:56:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8682 [2026-10-21 22:56:18][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,596 | MFU: 1.8% [2026-10-21 22:56:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8506 [2026-10-21 22:56:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3381 [2026-10-21 22:56:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9833 [2026-10-21 22:56:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2622 [2026-10-21 22:56:23][INFO][training.pretrain_engine] [Ph2|Step 405] loss=5.1028 | ppl=164.48 | aux=0.02506 | lr=2.98e-04 | gnorm=0.570 | tok/s=29,336 | MFU=7.0% | step=142975ms | tokens=1.227B | mem=4.2/32.7GB | cost=$33.20/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 09:21 (130.4h) | Phase3: SKIPPED | SFT: Oct 27 14:05 (135.2h) | DPO (Final): Oct 27 14:39 (135.7h) [2026-10-21 22:56:26][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step405 | Tokens: 1.227B | Spent: $33.20 | Remaining: $966.80 [2026-10-21 22:56:26][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step405 [2026-10-21 22:56:26][INFO][training.pretrain_engine] Starting global step 406 (accumulating 32 micro-batches)... [2026-10-21 22:56:26][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 22:56:26][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 22:56:26][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 22:56:26][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 22:56:26][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step405 to Phase-Technologies/forge-500m-pretrain (path: phase2_step405)... [2026-10-21 22:56:26][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 22:56:36][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 22:56:36][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step405 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 22:56:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0435 [2026-10-21 22:56:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0103 [2026-10-21 22:56:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0910 [2026-10-21 22:56:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9540 [2026-10-21 22:56:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9124 [2026-10-21 22:56:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5020 [2026-10-21 22:56:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5515 [2026-10-21 22:57:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0330 [2026-10-21 22:57:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0831 [2026-10-21 22:57:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1420 [2026-10-21 22:57:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5763 [2026-10-21 22:57:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1002 [2026-10-21 22:57:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6894 [2026-10-21 22:57:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5534 [2026-10-21 22:57:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8424 [2026-10-21 22:57:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0827 [2026-10-21 22:57:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1863 [2026-10-21 22:57:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1348 [2026-10-21 22:57:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1016 [2026-10-21 22:57:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2142 [2026-10-21 22:58:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9267 [2026-10-21 22:58:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8160 [2026-10-21 22:58:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.1449 [2026-10-21 22:58:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8455 [2026-10-21 22:58:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8048 [2026-10-21 22:58:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7393 [2026-10-21 22:58:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3643 [2026-10-21 22:58:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7108 [2026-10-21 22:58:44][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,582 | MFU: 1.8% [2026-10-21 22:58:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2057 [2026-10-21 22:58:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7875 [2026-10-21 22:58:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5448 [2026-10-21 22:58:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2422 [2026-10-21 22:58:49][INFO][training.pretrain_engine] [Ph2|Step 406] loss=5.0651 | ppl=158.40 | aux=0.02506 | lr=2.98e-04 | gnorm=0.582 | tok/s=29,338 | MFU=7.0% | step=142965ms | tokens=1.228B | mem=4.2/32.7GB | cost=$33.28/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 09:22 (130.4h) | Phase3: SKIPPED | SFT: Oct 27 14:06 (135.1h) | DPO (Final): Oct 27 14:40 (135.7h) [2026-10-21 22:58:49][INFO][training.pretrain_engine] Starting global step 407 (accumulating 32 micro-batches)... [2026-10-21 22:58:49][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 22:59:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7826 [2026-10-21 22:59:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3935 [2026-10-21 22:59:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2101 [2026-10-21 22:59:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1356 [2026-10-21 22:59:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8786 [2026-10-21 22:59:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3762 [2026-10-21 22:59:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2049 [2026-10-21 22:59:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2635 [2026-10-21 22:59:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8018 [2026-10-21 22:59:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0123 [2026-10-21 22:59:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8462 [2026-10-21 22:59:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3016 [2026-10-21 23:00:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0549 [2026-10-21 23:00:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6316 [2026-10-21 23:00:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4191 [2026-10-21 23:00:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8469 [2026-10-21 23:00:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5588 [2026-10-21 23:00:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1155 [2026-10-21 23:00:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8564 [2026-10-21 23:00:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3809 [2026-10-21 23:00:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1480 [2026-10-21 23:00:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2155 [2026-10-21 23:00:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6526 [2026-10-21 23:00:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3499 [2026-10-21 23:00:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1696 [2026-10-21 23:00:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5938 [2026-10-21 23:00:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7041 [2026-10-21 23:00:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7774 [2026-10-21 23:01:07][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,589 | MFU: 1.8% [2026-10-21 23:01:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9173 [2026-10-21 23:01:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3957 [2026-10-21 23:01:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2375 [2026-10-21 23:01:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2785 [2026-10-21 23:01:12][INFO][training.pretrain_engine] [Ph2|Step 407] loss=5.0950 | ppl=163.21 | aux=0.02506 | lr=2.98e-04 | gnorm=0.508 | tok/s=29,337 | MFU=7.0% | step=142971ms | tokens=1.229B | mem=4.2/32.7GB | cost=$33.36/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 09:24 (130.4h) | Phase3: SKIPPED | SFT: Oct 27 14:08 (135.1h) | DPO (Final): Oct 27 14:42 (135.7h) [2026-10-21 23:01:12][INFO][training.pretrain_engine] Starting global step 408 (accumulating 32 micro-batches)... [2026-10-21 23:01:12][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 23:01:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3225 [2026-10-21 23:01:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4522 [2026-10-21 23:01:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2515 [2026-10-21 23:01:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1167 [2026-10-21 23:01:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8134 [2026-10-21 23:01:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3764 [2026-10-21 23:01:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5294 [2026-10-21 23:01:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6413 [2026-10-21 23:02:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6856 [2026-10-21 23:02:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7784 [2026-10-21 23:02:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5615 [2026-10-21 23:02:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1148 [2026-10-21 23:02:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9434 [2026-10-21 23:02:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8938 [2026-10-21 23:02:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7465 [2026-10-21 23:02:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3267 [2026-10-21 23:02:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9676 [2026-10-21 23:02:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9743 [2026-10-21 23:02:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2034 [2026-10-21 23:02:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0019 [2026-10-21 23:02:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0108 [2026-10-21 23:02:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1970 [2026-10-21 23:02:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2424 [2026-10-21 23:02:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8240 [2026-10-21 23:03:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5857 [2026-10-21 23:03:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8349 [2026-10-21 23:03:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2750 [2026-10-21 23:03:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9888 [2026-10-21 23:03:30][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,591 | MFU: 1.8% [2026-10-21 23:03:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4766 [2026-10-21 23:03:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1413 [2026-10-21 23:03:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1358 [2026-10-21 23:03:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2203 [2026-10-21 23:03:35][INFO][training.pretrain_engine] [Ph2|Step 408] loss=4.9502 | ppl=141.20 | aux=0.02507 | lr=2.98e-04 | gnorm=0.637 | tok/s=29,336 | MFU=7.0% | step=142977ms | tokens=1.230B | mem=4.2/32.7GB | cost=$33.44/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 09:26 (130.4h) | Phase3: SKIPPED | SFT: Oct 27 14:10 (135.1h) | DPO (Final): Oct 27 14:45 (135.7h) [2026-10-21 23:03:35][INFO][training.pretrain_engine] Starting global step 409 (accumulating 32 micro-batches)... [2026-10-21 23:03:35][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 23:03:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4128 [2026-10-21 23:03:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3552 [2026-10-21 23:03:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4494 [2026-10-21 23:03:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3267 [2026-10-21 23:04:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9720 [2026-10-21 23:04:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2006 [2026-10-21 23:04:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5191 [2026-10-21 23:04:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0667 [2026-10-21 23:04:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3919 [2026-10-21 23:04:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8856 [2026-10-21 23:04:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1451 [2026-10-21 23:04:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0935 [2026-10-21 23:04:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8827 [2026-10-21 23:04:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2310 [2026-10-21 23:04:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7613 [2026-10-21 23:04:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9293 [2026-10-21 23:05:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7855 [2026-10-21 23:05:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8020 [2026-10-21 23:05:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0365 [2026-10-21 23:05:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0276 [2026-10-21 23:05:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9888 [2026-10-21 23:05:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0292 [2026-10-21 23:05:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0328 [2026-10-21 23:05:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9556 [2026-10-21 23:05:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6566 [2026-10-21 23:05:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5833 [2026-10-21 23:05:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7223 [2026-10-21 23:05:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1252 [2026-10-21 23:05:53][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,572 | MFU: 1.8% [2026-10-21 23:05:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2506 [2026-10-21 23:05:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0344 [2026-10-21 23:05:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0108 [2026-10-21 23:05:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0648 [2026-10-21 23:05:58][INFO][training.pretrain_engine] [Ph2|Step 409] loss=5.1298 | ppl=168.98 | aux=0.02506 | lr=2.98e-04 | gnorm=0.539 | tok/s=29,336 | MFU=7.0% | step=142976ms | tokens=1.231B | mem=4.2/32.7GB | cost=$33.52/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 09:28 (130.4h) | Phase3: SKIPPED | SFT: Oct 27 14:12 (135.1h) | DPO (Final): Oct 27 14:46 (135.7h) [2026-10-21 23:05:58][INFO][training.pretrain_engine] Starting global step 410 (accumulating 32 micro-batches)... [2026-10-21 23:05:58][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 23:06:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1235 [2026-10-21 23:06:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8292 [2026-10-21 23:06:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9710 [2026-10-21 23:06:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1519 [2026-10-21 23:06:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9773 [2026-10-21 23:06:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2138 [2026-10-21 23:06:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3543 [2026-10-21 23:06:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3291 [2026-10-21 23:06:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3455 [2026-10-21 23:06:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1981 [2026-10-21 23:06:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0989 [2026-10-21 23:06:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3457 [2026-10-21 23:07:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2725 [2026-10-21 23:07:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6416 [2026-10-21 23:07:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3265 [2026-10-21 23:07:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8227 [2026-10-21 23:07:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0558 [2026-10-21 23:07:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5382 [2026-10-21 23:07:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4145 [2026-10-21 23:07:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1202 [2026-10-21 23:07:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5510 [2026-10-21 23:07:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5896 [2026-10-21 23:07:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5332 [2026-10-21 23:07:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3098 [2026-10-21 23:08:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5833 [2026-10-21 23:08:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7217 [2026-10-21 23:08:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7760 [2026-10-21 23:08:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.1049 [2026-10-21 23:08:16][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,588 | MFU: 1.8% [2026-10-21 23:08:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3357 [2026-10-21 23:08:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2836 [2026-10-21 23:08:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2500 [2026-10-21 23:08:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1275 [2026-10-21 23:08:21][INFO][training.pretrain_engine] [Ph2|Step 410] loss=5.0755 | ppl=160.05 | aux=0.02506 | lr=2.98e-04 | gnorm=0.473 | tok/s=29,336 | MFU=7.0% | step=142973ms | tokens=1.232B | mem=4.2/32.7GB | cost=$33.61/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 09:30 (130.4h) | Phase3: SKIPPED | SFT: Oct 27 14:14 (135.1h) | DPO (Final): Oct 27 14:48 (135.7h) [2026-10-21 23:08:21][INFO][training.pretrain_engine] Starting global step 411 (accumulating 32 micro-batches)... [2026-10-21 23:08:21][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 23:08:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0825 [2026-10-21 23:08:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0752 [2026-10-21 23:08:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3988 [2026-10-21 23:08:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8129 [2026-10-21 23:08:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1965 [2026-10-21 23:08:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0935 [2026-10-21 23:08:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0768 [2026-10-21 23:08:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.5900 [2026-10-21 23:09:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9741 [2026-10-21 23:09:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7987 [2026-10-21 23:09:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6707 [2026-10-21 23:09:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0487 [2026-10-21 23:09:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7930 [2026-10-21 23:09:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8427 [2026-10-21 23:09:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9126 [2026-10-21 23:09:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9534 [2026-10-21 23:09:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1400 [2026-10-21 23:09:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2175 [2026-10-21 23:09:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9375 [2026-10-21 23:09:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8111 [2026-10-21 23:10:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1744 [2026-10-21 23:10:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3521 [2026-10-21 23:10:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0982 [2026-10-21 23:10:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2539 [2026-10-21 23:10:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1490 [2026-10-21 23:10:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4396 [2026-10-21 23:10:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4069 [2026-10-21 23:10:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9431 [2026-10-21 23:10:39][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,600 | MFU: 1.8% [2026-10-21 23:10:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4046 [2026-10-21 23:10:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3117 [2026-10-21 23:10:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4506 [2026-10-21 23:10:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4161 [2026-10-21 23:10:44][INFO][training.pretrain_engine] [Ph2|Step 411] loss=5.0510 | ppl=156.17 | aux=0.02505 | lr=2.98e-04 | gnorm=0.463 | tok/s=29,338 | MFU=7.0% | step=142966ms | tokens=1.233B | mem=4.2/32.7GB | cost=$33.69/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 09:31 (130.3h) | Phase3: SKIPPED | SFT: Oct 27 14:15 (135.1h) | DPO (Final): Oct 27 14:49 (135.7h) [2026-10-21 23:10:44][INFO][training.pretrain_engine] Starting global step 412 (accumulating 32 micro-batches)... [2026-10-21 23:10:44][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 23:11:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1607 [2026-10-21 23:11:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9564 [2026-10-21 23:11:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4510 [2026-10-21 23:11:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8497 [2026-10-21 23:11:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2132 [2026-10-21 23:11:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5449 [2026-10-21 23:11:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.1428 [2026-10-21 23:11:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4304 [2026-10-21 23:11:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0119 [2026-10-21 23:11:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0738 [2026-10-21 23:11:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8028 [2026-10-21 23:11:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1019 [2026-10-21 23:11:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9320 [2026-10-21 23:11:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7154 [2026-10-21 23:11:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2460 [2026-10-21 23:11:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6753 [2026-10-21 23:12:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2865 [2026-10-21 23:12:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5213 [2026-10-21 23:12:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1610 [2026-10-21 23:12:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8082 [2026-10-21 23:12:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9420 [2026-10-21 23:12:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3323 [2026-10-21 23:12:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9434 [2026-10-21 23:12:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8352 [2026-10-21 23:12:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1645 [2026-10-21 23:12:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9105 [2026-10-21 23:12:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1910 [2026-10-21 23:12:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3735 [2026-10-21 23:13:02][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,575 | MFU: 1.8% [2026-10-21 23:13:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9959 [2026-10-21 23:13:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3211 [2026-10-21 23:13:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6528 [2026-10-21 23:13:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1606 [2026-10-21 23:13:07][INFO][training.pretrain_engine] [Ph2|Step 412] loss=5.0350 | ppl=153.71 | aux=0.02507 | lr=2.98e-04 | gnorm=0.504 | tok/s=29,338 | MFU=7.0% | step=142964ms | tokens=1.234B | mem=4.2/32.7GB | cost=$33.77/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 09:33 (130.3h) | Phase3: SKIPPED | SFT: Oct 27 14:17 (135.1h) | DPO (Final): Oct 27 14:51 (135.6h) [2026-10-21 23:13:07][INFO][training.pretrain_engine] Starting global step 413 (accumulating 32 micro-batches)... [2026-10-21 23:13:07][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 23:13:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2124 [2026-10-21 23:13:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1250 [2026-10-21 23:13:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7491 [2026-10-21 23:13:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0688 [2026-10-21 23:13:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1985 [2026-10-21 23:13:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9403 [2026-10-21 23:13:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0817 [2026-10-21 23:13:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4356 [2026-10-21 23:14:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1041 [2026-10-21 23:14:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5049 [2026-10-21 23:14:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1620 [2026-10-21 23:14:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4283 [2026-10-21 23:14:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7756 [2026-10-21 23:14:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6615 [2026-10-21 23:14:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9461 [2026-10-21 23:14:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3504 [2026-10-21 23:14:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5954 [2026-10-21 23:14:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3621 [2026-10-21 23:14:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5454 [2026-10-21 23:14:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2109 [2026-10-21 23:14:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0402 [2026-10-21 23:14:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7041 [2026-10-21 23:14:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5815 [2026-10-21 23:14:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7494 [2026-10-21 23:15:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4142 [2026-10-21 23:15:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1894 [2026-10-21 23:15:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5355 [2026-10-21 23:15:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7860 [2026-10-21 23:15:24][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,610 | MFU: 1.8% [2026-10-21 23:15:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6002 [2026-10-21 23:15:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9137 [2026-10-21 23:15:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3580 [2026-10-21 23:15:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2974 [2026-10-21 23:15:30][INFO][training.pretrain_engine] [Ph2|Step 413] loss=5.0617 | ppl=157.86 | aux=0.02506 | lr=2.98e-04 | gnorm=0.486 | tok/s=29,339 | MFU=7.0% | step=142959ms | tokens=1.235B | mem=4.2/32.7GB | cost=$33.85/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 09:34 (130.3h) | Phase3: SKIPPED | SFT: Oct 27 14:18 (135.1h) | DPO (Final): Oct 27 14:52 (135.6h) [2026-10-21 23:15:30][INFO][training.pretrain_engine] Starting global step 414 (accumulating 32 micro-batches)... [2026-10-21 23:15:30][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 23:15:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0512 [2026-10-21 23:15:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5816 [2026-10-21 23:15:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3456 [2026-10-21 23:15:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1245 [2026-10-21 23:16:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3664 [2026-10-21 23:16:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7744 [2026-10-21 23:16:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7129 [2026-10-21 23:16:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0492 [2026-10-21 23:16:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9528 [2026-10-21 23:16:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0964 [2026-10-21 23:16:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1374 [2026-10-21 23:16:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8162 [2026-10-21 23:16:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2717 [2026-10-21 23:16:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9620 [2026-10-21 23:16:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1703 [2026-10-21 23:16:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0668 [2026-10-21 23:16:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4766 [2026-10-21 23:16:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7826 [2026-10-21 23:16:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7886 [2026-10-21 23:16:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9813 [2026-10-21 23:17:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8733 [2026-10-21 23:17:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0348 [2026-10-21 23:17:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2548 [2026-10-21 23:17:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8280 [2026-10-21 23:17:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9310 [2026-10-21 23:17:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8177 [2026-10-21 23:17:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7565 [2026-10-21 23:17:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9429 [2026-10-21 23:17:47][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,607 | MFU: 1.8% [2026-10-21 23:17:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0386 [2026-10-21 23:17:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9674 [2026-10-21 23:17:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4558 [2026-10-21 23:17:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6239 [2026-10-21 23:17:52][INFO][training.pretrain_engine] [Ph2|Step 414] loss=5.0283 | ppl=152.67 | aux=0.02506 | lr=2.98e-04 | gnorm=0.484 | tok/s=29,341 | MFU=7.0% | step=142951ms | tokens=1.236B | mem=4.2/32.7GB | cost=$33.93/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 09:36 (130.3h) | Phase3: SKIPPED | SFT: Oct 27 14:20 (135.0h) | DPO (Final): Oct 27 14:54 (135.6h) [2026-10-21 23:17:55][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step414 | Tokens: 1.236B | Spent: $33.94 | Remaining: $966.06 [2026-10-21 23:17:55][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step414 [2026-10-21 23:17:55][INFO][training.pretrain_engine] Starting global step 415 (accumulating 32 micro-batches)... [2026-10-21 23:17:55][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 23:17:55][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 23:17:55][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 23:17:55][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 23:17:55][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step414 to Phase-Technologies/forge-500m-pretrain (path: phase2_step414)... [2026-10-21 23:17:55][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 23:18:06][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 23:18:06][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step414 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 23:18:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5925 [2026-10-21 23:18:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7096 [2026-10-21 23:18:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5663 [2026-10-21 23:18:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5537 [2026-10-21 23:18:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6934 [2026-10-21 23:18:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8061 [2026-10-21 23:18:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9790 [2026-10-21 23:18:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9518 [2026-10-21 23:18:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3143 [2026-10-21 23:18:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1285 [2026-10-21 23:18:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1265 [2026-10-21 23:18:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1317 [2026-10-21 23:19:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0774 [2026-10-21 23:19:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9535 [2026-10-21 23:19:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8488 [2026-10-21 23:19:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9603 [2026-10-21 23:19:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2559 [2026-10-21 23:19:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2372 [2026-10-21 23:19:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2054 [2026-10-21 23:19:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2379 [2026-10-21 23:19:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=3.6806 [2026-10-21 23:19:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8429 [2026-10-21 23:19:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6585 [2026-10-21 23:19:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7686 [2026-10-21 23:19:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3006 [2026-10-21 23:19:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2165 [2026-10-21 23:19:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0821 [2026-10-21 23:20:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8368 [2026-10-21 23:20:13][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,604 | MFU: 1.8% [2026-10-21 23:20:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1042 [2026-10-21 23:20:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0133 [2026-10-21 23:20:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2016 [2026-10-21 23:20:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0664 [2026-10-21 23:20:18][INFO][training.pretrain_engine] [Ph2|Step 415] loss=5.0781 | ppl=160.46 | aux=0.02507 | lr=2.98e-04 | gnorm=0.479 | tok/s=29,342 | MFU=7.0% | step=142945ms | tokens=1.237B | mem=4.2/32.7GB | cost=$34.02/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 09:37 (130.3h) | Phase3: SKIPPED | SFT: Oct 27 14:21 (135.0h) | DPO (Final): Oct 27 14:55 (135.6h) [2026-10-21 23:20:18][INFO][training.pretrain_engine] Starting global step 416 (accumulating 32 micro-batches)... [2026-10-21 23:20:18][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 23:20:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1453 [2026-10-21 23:20:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0830 [2026-10-21 23:20:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4628 [2026-10-21 23:20:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8800 [2026-10-21 23:20:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4616 [2026-10-21 23:20:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0381 [2026-10-21 23:20:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7026 [2026-10-21 23:20:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6807 [2026-10-21 23:21:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2425 [2026-10-21 23:21:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2679 [2026-10-21 23:21:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3404 [2026-10-21 23:21:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5909 [2026-10-21 23:21:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8915 [2026-10-21 23:21:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2216 [2026-10-21 23:21:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3361 [2026-10-21 23:21:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7846 [2026-10-21 23:21:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1791 [2026-10-21 23:21:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6539 [2026-10-21 23:21:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7623 [2026-10-21 23:21:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5907 [2026-10-21 23:22:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1845 [2026-10-21 23:22:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7439 [2026-10-21 23:22:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0365 [2026-10-21 23:22:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5278 [2026-10-21 23:22:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0865 [2026-10-21 23:22:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0333 [2026-10-21 23:22:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1215 [2026-10-21 23:22:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9731 [2026-10-21 23:22:35][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,602 | MFU: 1.8% [2026-10-21 23:22:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8779 [2026-10-21 23:22:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1322 [2026-10-21 23:22:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6133 [2026-10-21 23:22:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.0297 [2026-10-21 23:22:40][INFO][training.pretrain_engine] [Ph2|Step 416] loss=5.0198 | ppl=151.38 | aux=0.02507 | lr=2.98e-04 | gnorm=0.479 | tok/s=29,342 | MFU=7.0% | step=142947ms | tokens=1.238B | mem=4.2/32.7GB | cost=$34.10/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 09:39 (130.3h) | Phase3: SKIPPED | SFT: Oct 27 14:23 (135.0h) | DPO (Final): Oct 27 14:57 (135.6h) [2026-10-21 23:22:40][INFO][training.pretrain_engine] Starting global step 417 (accumulating 32 micro-batches)... [2026-10-21 23:22:40][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 23:22:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1180 [2026-10-21 23:22:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9260 [2026-10-21 23:22:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9916 [2026-10-21 23:22:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9423 [2026-10-21 23:23:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7594 [2026-10-21 23:23:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9470 [2026-10-21 23:23:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9916 [2026-10-21 23:23:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3471 [2026-10-21 23:23:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0998 [2026-10-21 23:23:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9277 [2026-10-21 23:23:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8719 [2026-10-21 23:23:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9355 [2026-10-21 23:23:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7968 [2026-10-21 23:23:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3690 [2026-10-21 23:23:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4448 [2026-10-21 23:23:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8100 [2026-10-21 23:24:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9101 [2026-10-21 23:24:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7633 [2026-10-21 23:24:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9389 [2026-10-21 23:24:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5993 [2026-10-21 23:24:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6429 [2026-10-21 23:24:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4591 [2026-10-21 23:24:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3693 [2026-10-21 23:24:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8003 [2026-10-21 23:24:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7244 [2026-10-21 23:24:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3051 [2026-10-21 23:24:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5720 [2026-10-21 23:24:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8243 [2026-10-21 23:24:59][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,587 | MFU: 1.8% [2026-10-21 23:25:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3821 [2026-10-21 23:25:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6388 [2026-10-21 23:25:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5975 [2026-10-21 23:25:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4650 [2026-10-21 23:25:03][INFO][training.pretrain_engine] [Ph2|Step 417] loss=4.9990 | ppl=148.26 | aux=0.02505 | lr=2.98e-04 | gnorm=0.488 | tok/s=29,342 | MFU=7.0% | step=142945ms | tokens=1.239B | mem=4.2/32.7GB | cost=$34.18/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 09:41 (130.3h) | Phase3: SKIPPED | SFT: Oct 27 14:25 (135.0h) | DPO (Final): Oct 27 14:59 (135.6h) [2026-10-21 23:25:03][INFO][training.pretrain_engine] Starting global step 418 (accumulating 32 micro-batches)... [2026-10-21 23:25:03][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 23:25:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1408 [2026-10-21 23:25:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0060 [2026-10-21 23:25:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0525 [2026-10-21 23:25:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5201 [2026-10-21 23:25:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1876 [2026-10-21 23:25:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7112 [2026-10-21 23:25:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3705 [2026-10-21 23:25:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4056 [2026-10-21 23:25:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6932 [2026-10-21 23:25:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7480 [2026-10-21 23:25:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8547 [2026-10-21 23:25:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6063 [2026-10-21 23:26:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0094 [2026-10-21 23:26:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6940 [2026-10-21 23:26:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2775 [2026-10-21 23:26:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9492 [2026-10-21 23:26:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0917 [2026-10-21 23:26:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.1951 [2026-10-21 23:26:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3087 [2026-10-21 23:26:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9786 [2026-10-21 23:26:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9024 [2026-10-21 23:26:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0075 [2026-10-21 23:26:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3082 [2026-10-21 23:26:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2436 [2026-10-21 23:27:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4237 [2026-10-21 23:27:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1390 [2026-10-21 23:27:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9752 [2026-10-21 23:27:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1820 [2026-10-21 23:27:22][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,589 | MFU: 1.8% [2026-10-21 23:27:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4254 [2026-10-21 23:27:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9666 [2026-10-21 23:27:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1941 [2026-10-21 23:27:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1388 [2026-10-21 23:27:27][INFO][training.pretrain_engine] [Ph2|Step 418] loss=4.9526 | ppl=141.54 | aux=0.02506 | lr=2.98e-04 | gnorm=0.520 | tok/s=29,342 | MFU=7.0% | step=142946ms | tokens=1.240B | mem=4.2/32.7GB | cost=$34.26/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 09:43 (130.3h) | Phase3: SKIPPED | SFT: Oct 27 14:27 (135.0h) | DPO (Final): Oct 27 15:01 (135.6h) [2026-10-21 23:27:27][INFO][training.pretrain_engine] Starting global step 419 (accumulating 32 micro-batches)... [2026-10-21 23:27:27][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 23:27:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7639 [2026-10-21 23:27:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9916 [2026-10-21 23:27:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8827 [2026-10-21 23:27:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7808 [2026-10-21 23:28:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7134 [2026-10-21 23:28:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9820 [2026-10-21 23:28:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6179 [2026-10-21 23:28:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9614 [2026-10-21 23:28:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.1768 [2026-10-21 23:28:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4578 [2026-10-21 23:28:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5964 [2026-10-21 23:28:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6273 [2026-10-21 23:28:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8103 [2026-10-21 23:28:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1141 [2026-10-21 23:28:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9939 [2026-10-21 23:28:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5504 [2026-10-21 23:28:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8640 [2026-10-21 23:28:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6950 [2026-10-21 23:28:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4018 [2026-10-21 23:28:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5825 [2026-10-21 23:29:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1665 [2026-10-21 23:29:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4217 [2026-10-21 23:29:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0944 [2026-10-21 23:29:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0961 [2026-10-21 23:29:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8213 [2026-10-21 23:29:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9578 [2026-10-21 23:29:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1990 [2026-10-21 23:29:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1621 [2026-10-21 23:29:44][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,619 | MFU: 1.8% [2026-10-21 23:29:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5944 [2026-10-21 23:29:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5121 [2026-10-21 23:29:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5465 [2026-10-21 23:29:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7558 [2026-10-21 23:29:49][INFO][training.pretrain_engine] [Ph2|Step 419] loss=4.9481 | ppl=140.90 | aux=0.02505 | lr=2.98e-04 | gnorm=0.520 | tok/s=29,343 | MFU=7.0% | step=142943ms | tokens=1.242B | mem=4.2/32.7GB | cost=$34.34/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 09:44 (130.2h) | Phase3: SKIPPED | SFT: Oct 27 14:28 (135.0h) | DPO (Final): Oct 27 15:02 (135.5h) [2026-10-21 23:29:49][INFO][training.pretrain_engine] Starting global step 420 (accumulating 32 micro-batches)... [2026-10-21 23:29:49][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 23:30:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7740 [2026-10-21 23:30:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8543 [2026-10-21 23:30:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8362 [2026-10-21 23:30:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5449 [2026-10-21 23:30:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8856 [2026-10-21 23:30:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0869 [2026-10-21 23:30:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1871 [2026-10-21 23:30:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0792 [2026-10-21 23:30:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8997 [2026-10-21 23:30:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1845 [2026-10-21 23:30:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1668 [2026-10-21 23:30:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7547 [2026-10-21 23:31:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9039 [2026-10-21 23:31:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0816 [2026-10-21 23:31:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3314 [2026-10-21 23:31:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1087 [2026-10-21 23:31:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0022 [2026-10-21 23:31:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1258 [2026-10-21 23:31:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9165 [2026-10-21 23:31:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0452 [2026-10-21 23:31:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3995 [2026-10-21 23:31:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3031 [2026-10-21 23:31:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0532 [2026-10-21 23:31:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2115 [2026-10-21 23:31:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5749 [2026-10-21 23:31:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9246 [2026-10-21 23:31:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4997 [2026-10-21 23:31:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7951 [2026-10-21 23:32:07][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,594 | MFU: 1.8% [2026-10-21 23:32:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5718 [2026-10-21 23:32:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2270 [2026-10-21 23:32:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1068 [2026-10-21 23:32:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1521 [2026-10-21 23:32:12][INFO][training.pretrain_engine] [Ph2|Step 420] loss=5.0560 | ppl=156.96 | aux=0.02506 | lr=2.98e-04 | gnorm=0.430 | tok/s=29,342 | MFU=7.0% | step=142945ms | tokens=1.243B | mem=4.2/32.7GB | cost=$34.43/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 09:46 (130.2h) | Phase3: SKIPPED | SFT: Oct 27 14:30 (135.0h) | DPO (Final): Oct 27 15:04 (135.5h) [2026-10-21 23:32:12][INFO][training.pretrain_engine] Starting global step 421 (accumulating 32 micro-batches)... [2026-10-21 23:32:12][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 23:32:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4666 [2026-10-21 23:32:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3395 [2026-10-21 23:32:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6091 [2026-10-21 23:32:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9724 [2026-10-21 23:32:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0757 [2026-10-21 23:32:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1054 [2026-10-21 23:32:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0791 [2026-10-21 23:32:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0815 [2026-10-21 23:33:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8267 [2026-10-21 23:33:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0813 [2026-10-21 23:33:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9330 [2026-10-21 23:33:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7117 [2026-10-21 23:33:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9256 [2026-10-21 23:33:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5297 [2026-10-21 23:33:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2773 [2026-10-21 23:33:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0343 [2026-10-21 23:33:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3042 [2026-10-21 23:33:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.0574 [2026-10-21 23:33:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=3.9704 [2026-10-21 23:33:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6714 [2026-10-21 23:33:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3146 [2026-10-21 23:33:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7534 [2026-10-21 23:34:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8598 [2026-10-21 23:34:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8718 [2026-10-21 23:34:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5532 [2026-10-21 23:34:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2333 [2026-10-21 23:34:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5973 [2026-10-21 23:34:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1779 [2026-10-21 23:34:31][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,590 | MFU: 1.8% [2026-10-21 23:34:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5597 [2026-10-21 23:34:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0839 [2026-10-21 23:34:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6458 [2026-10-21 23:34:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7572 [2026-10-21 23:34:36][INFO][training.pretrain_engine] [Ph2|Step 421] loss=5.0585 | ppl=157.36 | aux=0.02507 | lr=2.98e-04 | gnorm=0.598 | tok/s=29,341 | MFU=7.0% | step=142951ms | tokens=1.244B | mem=4.2/32.7GB | cost=$34.51/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 09:48 (130.2h) | Phase3: SKIPPED | SFT: Oct 27 14:32 (135.0h) | DPO (Final): Oct 27 15:06 (135.5h) [2026-10-21 23:34:36][INFO][training.pretrain_engine] Starting global step 422 (accumulating 32 micro-batches)... [2026-10-21 23:34:36][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 23:34:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2400 [2026-10-21 23:34:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7998 [2026-10-21 23:34:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2263 [2026-10-21 23:34:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3120 [2026-10-21 23:35:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3433 [2026-10-21 23:35:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3071 [2026-10-21 23:35:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.7598 [2026-10-21 23:35:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2866 [2026-10-21 23:35:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6642 [2026-10-21 23:35:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0926 [2026-10-21 23:35:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3489 [2026-10-21 23:35:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2273 [2026-10-21 23:35:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4781 [2026-10-21 23:35:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7317 [2026-10-21 23:35:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2041 [2026-10-21 23:35:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9981 [2026-10-21 23:36:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2730 [2026-10-21 23:36:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0607 [2026-10-21 23:36:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1693 [2026-10-21 23:36:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3706 [2026-10-21 23:36:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7484 [2026-10-21 23:36:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6457 [2026-10-21 23:36:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7018 [2026-10-21 23:36:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7424 [2026-10-21 23:36:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0597 [2026-10-21 23:36:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7107 [2026-10-21 23:36:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8372 [2026-10-21 23:36:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0972 [2026-10-21 23:36:54][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,593 | MFU: 1.8% [2026-10-21 23:36:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5364 [2026-10-21 23:36:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8385 [2026-10-21 23:36:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0803 [2026-10-21 23:36:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0453 [2026-10-21 23:37:00][INFO][training.pretrain_engine] [Ph2|Step 422] loss=4.9726 | ppl=144.40 | aux=0.02505 | lr=2.98e-04 | gnorm=0.742 | tok/s=29,340 | MFU=7.0% | step=142954ms | tokens=1.245B | mem=4.2/32.7GB | cost=$34.59/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 09:50 (130.2h) | Phase3: SKIPPED | SFT: Oct 27 14:34 (135.0h) | DPO (Final): Oct 27 15:08 (135.5h) [2026-10-21 23:37:00][INFO][training.pretrain_engine] Starting global step 423 (accumulating 32 micro-batches)... [2026-10-21 23:37:00][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 23:37:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0359 [2026-10-21 23:37:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2322 [2026-10-21 23:37:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5099 [2026-10-21 23:37:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4128 [2026-10-21 23:37:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6804 [2026-10-21 23:37:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0280 [2026-10-21 23:37:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4113 [2026-10-21 23:37:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3305 [2026-10-21 23:37:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0898 [2026-10-21 23:37:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8303 [2026-10-21 23:37:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0071 [2026-10-21 23:37:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1485 [2026-10-21 23:38:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0756 [2026-10-21 23:38:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1042 [2026-10-21 23:38:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4403 [2026-10-21 23:38:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0024 [2026-10-21 23:38:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7075 [2026-10-21 23:38:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0154 [2026-10-21 23:38:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9303 [2026-10-21 23:38:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7539 [2026-10-21 23:38:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.0746 [2026-10-21 23:38:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4283 [2026-10-21 23:38:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3630 [2026-10-21 23:38:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3233 [2026-10-21 23:39:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2683 [2026-10-21 23:39:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4072 [2026-10-21 23:39:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0663 [2026-10-21 23:39:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4977 [2026-10-21 23:39:18][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,562 | MFU: 1.8% [2026-10-21 23:39:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3977 [2026-10-21 23:39:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4471 [2026-10-21 23:39:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2039 [2026-10-21 23:39:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0943 [2026-10-21 23:39:23][INFO][training.pretrain_engine] [Ph2|Step 423] loss=5.0182 | ppl=151.14 | aux=0.02506 | lr=2.98e-04 | gnorm=0.918 | tok/s=29,339 | MFU=7.0% | step=142960ms | tokens=1.246B | mem=4.2/32.7GB | cost=$34.67/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 09:52 (130.2h) | Phase3: SKIPPED | SFT: Oct 27 14:36 (135.0h) | DPO (Final): Oct 27 15:10 (135.5h) [2026-10-21 23:39:27][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step423 | Tokens: 1.246B | Spent: $34.67 | Remaining: $965.33 [2026-10-21 23:39:27][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step423 [2026-10-21 23:39:27][INFO][training.pretrain_engine] Starting global step 424 (accumulating 32 micro-batches)... [2026-10-21 23:39:27][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 23:39:27][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-21 23:39:27][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-21 23:39:27][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-21 23:39:27][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step423 to Phase-Technologies/forge-500m-pretrain (path: phase2_step423)... [2026-10-21 23:39:27][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-21 23:39:39][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-21 23:39:39][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step423 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-21 23:39:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9759 [2026-10-21 23:39:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7929 [2026-10-21 23:39:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9334 [2026-10-21 23:39:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6171 [2026-10-21 23:39:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7273 [2026-10-21 23:39:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7122 [2026-10-21 23:39:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7507 [2026-10-21 23:40:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7663 [2026-10-21 23:40:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0075 [2026-10-21 23:40:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5711 [2026-10-21 23:40:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0289 [2026-10-21 23:40:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1291 [2026-10-21 23:40:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0851 [2026-10-21 23:40:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8206 [2026-10-21 23:40:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8646 [2026-10-21 23:40:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8066 [2026-10-21 23:40:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1112 [2026-10-21 23:40:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.2757 [2026-10-21 23:40:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8514 [2026-10-21 23:40:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2223 [2026-10-21 23:41:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2474 [2026-10-21 23:41:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1733 [2026-10-21 23:41:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2657 [2026-10-21 23:41:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7347 [2026-10-21 23:41:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9262 [2026-10-21 23:41:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0419 [2026-10-21 23:41:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7874 [2026-10-21 23:41:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1054 [2026-10-21 23:41:45][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,586 | MFU: 1.8% [2026-10-21 23:41:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7508 [2026-10-21 23:41:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9228 [2026-10-21 23:41:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3061 [2026-10-21 23:41:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0122 [2026-10-21 23:41:49][INFO][training.pretrain_engine] [Ph2|Step 424] loss=5.0171 | ppl=150.97 | aux=0.02506 | lr=2.98e-04 | gnorm=1.031 | tok/s=29,341 | MFU=7.0% | step=142952ms | tokens=1.247B | mem=4.2/32.7GB | cost=$34.76/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 09:54 (130.2h) | Phase3: SKIPPED | SFT: Oct 27 14:38 (134.9h) | DPO (Final): Oct 27 15:12 (135.5h) [2026-10-21 23:41:49][INFO][training.pretrain_engine] Starting global step 425 (accumulating 32 micro-batches)... [2026-10-21 23:41:49][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 23:42:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.9301 [2026-10-21 23:42:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4195 [2026-10-21 23:42:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5822 [2026-10-21 23:42:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6726 [2026-10-21 23:42:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4397 [2026-10-21 23:42:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0173 [2026-10-21 23:42:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4638 [2026-10-21 23:42:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8461 [2026-10-21 23:42:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8269 [2026-10-21 23:42:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2420 [2026-10-21 23:42:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9893 [2026-10-21 23:42:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0686 [2026-10-21 23:43:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1684 [2026-10-21 23:43:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0223 [2026-10-21 23:43:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8769 [2026-10-21 23:43:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9032 [2026-10-21 23:43:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5452 [2026-10-21 23:43:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7052 [2026-10-21 23:43:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2595 [2026-10-21 23:43:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4973 [2026-10-21 23:43:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2905 [2026-10-21 23:43:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4626 [2026-10-21 23:43:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2529 [2026-10-21 23:43:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1533 [2026-10-21 23:43:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8354 [2026-10-21 23:43:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4042 [2026-10-21 23:43:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1136 [2026-10-21 23:43:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5367 [2026-10-21 23:44:08][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,583 | MFU: 1.8% [2026-10-21 23:44:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6780 [2026-10-21 23:44:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1101 [2026-10-21 23:44:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1194 [2026-10-21 23:44:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0247 [2026-10-21 23:44:13][INFO][training.pretrain_engine] [Ph2|Step 425] loss=5.1329 | ppl=169.51 | aux=0.02507 | lr=2.98e-04 | gnorm=0.875 | tok/s=29,338 | MFU=7.0% | step=142963ms | tokens=1.248B | mem=4.2/32.7GB | cost=$34.84/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 09:56 (130.2h) | Phase3: SKIPPED | SFT: Oct 27 14:40 (134.9h) | DPO (Final): Oct 27 15:14 (135.5h) [2026-10-21 23:44:13][INFO][training.pretrain_engine] Starting global step 426 (accumulating 32 micro-batches)... [2026-10-21 23:44:13][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 23:44:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4872 [2026-10-21 23:44:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1403 [2026-10-21 23:44:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0467 [2026-10-21 23:44:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2966 [2026-10-21 23:44:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1169 [2026-10-21 23:44:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1142 [2026-10-21 23:44:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9142 [2026-10-21 23:44:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7971 [2026-10-21 23:45:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2444 [2026-10-21 23:45:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4125 [2026-10-21 23:45:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8169 [2026-10-21 23:45:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6796 [2026-10-21 23:45:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4963 [2026-10-21 23:45:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2202 [2026-10-21 23:45:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2232 [2026-10-21 23:45:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8128 [2026-10-21 23:45:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4227 [2026-10-21 23:45:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0351 [2026-10-21 23:45:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0515 [2026-10-21 23:45:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0320 [2026-10-21 23:45:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=3.9177 [2026-10-21 23:46:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1906 [2026-10-21 23:46:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9367 [2026-10-21 23:46:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8807 [2026-10-21 23:46:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2277 [2026-10-21 23:46:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0284 [2026-10-21 23:46:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0057 [2026-10-21 23:46:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7428 [2026-10-21 23:46:31][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,597 | MFU: 1.8% [2026-10-21 23:46:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0530 [2026-10-21 23:46:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7987 [2026-10-21 23:46:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9079 [2026-10-21 23:46:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3420 [2026-10-21 23:46:36][INFO][training.pretrain_engine] [Ph2|Step 426] loss=5.1240 | ppl=168.01 | aux=0.02507 | lr=2.98e-04 | gnorm=0.832 | tok/s=29,338 | MFU=7.0% | step=142964ms | tokens=1.249B | mem=4.2/32.7GB | cost=$34.92/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 09:58 (130.2h) | Phase3: SKIPPED | SFT: Oct 27 14:42 (134.9h) | DPO (Final): Oct 27 15:16 (135.5h) [2026-10-21 23:46:36][INFO][training.pretrain_engine] Starting global step 427 (accumulating 32 micro-batches)... [2026-10-21 23:46:36][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 23:46:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0724 [2026-10-21 23:46:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1636 [2026-10-21 23:46:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5680 [2026-10-21 23:46:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5443 [2026-10-21 23:47:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3649 [2026-10-21 23:47:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9364 [2026-10-21 23:47:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1070 [2026-10-21 23:47:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3334 [2026-10-21 23:47:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2640 [2026-10-21 23:47:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5371 [2026-10-21 23:47:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2555 [2026-10-21 23:47:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0323 [2026-10-21 23:47:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2825 [2026-10-21 23:47:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2019 [2026-10-21 23:47:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7421 [2026-10-21 23:47:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6210 [2026-10-21 23:48:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3659 [2026-10-21 23:48:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0913 [2026-10-21 23:48:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2931 [2026-10-21 23:48:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6910 [2026-10-21 23:48:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0425 [2026-10-21 23:48:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3332 [2026-10-21 23:48:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3337 [2026-10-21 23:48:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3406 [2026-10-21 23:48:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5073 [2026-10-21 23:48:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4180 [2026-10-21 23:48:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1424 [2026-10-21 23:48:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4638 [2026-10-21 23:48:54][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,609 | MFU: 1.8% [2026-10-21 23:49:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5744 [2026-10-21 23:49:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3055 [2026-10-21 23:49:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4314 [2026-10-21 23:49:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3977 [2026-10-21 23:49:00][INFO][training.pretrain_engine] [Ph2|Step 427] loss=5.0371 | ppl=154.02 | aux=0.02506 | lr=2.98e-04 | gnorm=0.684 | tok/s=29,337 | MFU=7.0% | step=142967ms | tokens=1.250B | mem=4.2/32.7GB | cost=$35.00/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 10:00 (130.2h) | Phase3: SKIPPED | SFT: Oct 27 14:44 (134.9h) | DPO (Final): Oct 27 15:18 (135.5h) [2026-10-21 23:49:00][INFO][training.pretrain_engine] Starting global step 428 (accumulating 32 micro-batches)... [2026-10-21 23:49:00][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 23:49:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7061 [2026-10-21 23:49:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7288 [2026-10-21 23:49:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.6443 [2026-10-21 23:49:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7823 [2026-10-21 23:49:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4740 [2026-10-21 23:49:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.9870 [2026-10-21 23:49:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9151 [2026-10-21 23:49:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9418 [2026-10-21 23:49:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7175 [2026-10-21 23:49:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3810 [2026-10-21 23:49:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3746 [2026-10-21 23:49:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3689 [2026-10-21 23:50:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7405 [2026-10-21 23:50:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7657 [2026-10-21 23:50:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3267 [2026-10-21 23:50:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6984 [2026-10-21 23:50:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0477 [2026-10-21 23:50:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1355 [2026-10-21 23:50:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0124 [2026-10-21 23:50:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0206 [2026-10-21 23:50:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2423 [2026-10-21 23:50:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3996 [2026-10-21 23:50:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9484 [2026-10-21 23:50:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6197 [2026-10-21 23:51:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8705 [2026-10-21 23:51:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7205 [2026-10-21 23:51:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6368 [2026-10-21 23:51:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7331 [2026-10-21 23:51:18][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,587 | MFU: 1.8% [2026-10-21 23:51:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3054 [2026-10-21 23:51:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7538 [2026-10-21 23:51:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7587 [2026-10-21 23:51:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6765 [2026-10-21 23:51:23][INFO][training.pretrain_engine] [Ph2|Step 428] loss=4.9422 | ppl=140.08 | aux=0.02507 | lr=2.98e-04 | gnorm=0.715 | tok/s=29,337 | MFU=7.0% | step=142970ms | tokens=1.251B | mem=4.2/32.7GB | cost=$35.08/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 10:02 (130.2h) | Phase3: SKIPPED | SFT: Oct 27 14:46 (134.9h) | DPO (Final): Oct 27 15:20 (135.5h) [2026-10-21 23:51:23][INFO][training.pretrain_engine] Starting global step 429 (accumulating 32 micro-batches)... [2026-10-21 23:51:23][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 23:51:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2179 [2026-10-21 23:51:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5494 [2026-10-21 23:51:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1908 [2026-10-21 23:51:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1167 [2026-10-21 23:51:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7533 [2026-10-21 23:51:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9790 [2026-10-21 23:51:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7513 [2026-10-21 23:51:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5028 [2026-10-21 23:52:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9089 [2026-10-21 23:52:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9077 [2026-10-21 23:52:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7164 [2026-10-21 23:52:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0262 [2026-10-21 23:52:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8180 [2026-10-21 23:52:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2316 [2026-10-21 23:52:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0513 [2026-10-21 23:52:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8339 [2026-10-21 23:52:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6144 [2026-10-21 23:52:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4790 [2026-10-21 23:52:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7619 [2026-10-21 23:52:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6111 [2026-10-21 23:53:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3713 [2026-10-21 23:53:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9462 [2026-10-21 23:53:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5545 [2026-10-21 23:53:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0769 [2026-10-21 23:53:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3293 [2026-10-21 23:53:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3632 [2026-10-21 23:53:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0773 [2026-10-21 23:53:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0384 [2026-10-21 23:53:41][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,591 | MFU: 1.8% [2026-10-21 23:53:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9406 [2026-10-21 23:53:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8465 [2026-10-21 23:53:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2634 [2026-10-21 23:53:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2354 [2026-10-21 23:53:46][INFO][training.pretrain_engine] [Ph2|Step 429] loss=5.0811 | ppl=160.95 | aux=0.02505 | lr=2.98e-04 | gnorm=0.602 | tok/s=29,338 | MFU=7.0% | step=142964ms | tokens=1.252B | mem=4.2/32.7GB | cost=$35.17/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 10:03 (130.2h) | Phase3: SKIPPED | SFT: Oct 27 14:47 (134.9h) | DPO (Final): Oct 27 15:22 (135.5h) [2026-10-21 23:53:46][INFO][training.pretrain_engine] Starting global step 430 (accumulating 32 micro-batches)... [2026-10-21 23:53:46][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 23:54:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3283 [2026-10-21 23:54:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0996 [2026-10-21 23:54:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3122 [2026-10-21 23:54:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9822 [2026-10-21 23:54:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9787 [2026-10-21 23:54:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3667 [2026-10-21 23:54:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6758 [2026-10-21 23:54:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7154 [2026-10-21 23:54:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1314 [2026-10-21 23:54:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2607 [2026-10-21 23:54:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8076 [2026-10-21 23:54:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9706 [2026-10-21 23:54:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2532 [2026-10-21 23:54:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9272 [2026-10-21 23:54:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8831 [2026-10-21 23:54:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7101 [2026-10-21 23:55:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2913 [2026-10-21 23:55:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8024 [2026-10-21 23:55:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9163 [2026-10-21 23:55:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0534 [2026-10-21 23:55:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2702 [2026-10-21 23:55:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6044 [2026-10-21 23:55:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5368 [2026-10-21 23:55:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4702 [2026-10-21 23:55:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7556 [2026-10-21 23:55:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8882 [2026-10-21 23:55:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1443 [2026-10-21 23:55:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0169 [2026-10-21 23:56:05][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,583 | MFU: 1.8% [2026-10-21 23:56:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9073 [2026-10-21 23:56:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7035 [2026-10-21 23:56:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7471 [2026-10-21 23:56:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1235 [2026-10-21 23:56:10][INFO][training.pretrain_engine] [Ph2|Step 430] loss=5.0456 | ppl=155.33 | aux=0.02506 | lr=2.98e-04 | gnorm=0.664 | tok/s=29,338 | MFU=7.0% | step=142966ms | tokens=1.253B | mem=4.2/32.7GB | cost=$35.25/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 10:05 (130.2h) | Phase3: SKIPPED | SFT: Oct 27 14:49 (134.9h) | DPO (Final): Oct 27 15:23 (135.5h) [2026-10-21 23:56:10][INFO][training.pretrain_engine] Starting global step 431 (accumulating 32 micro-batches)... [2026-10-21 23:56:10][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 23:56:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4530 [2026-10-21 23:56:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9785 [2026-10-21 23:56:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2073 [2026-10-21 23:56:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2817 [2026-10-21 23:56:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3708 [2026-10-21 23:56:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9613 [2026-10-21 23:56:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9685 [2026-10-21 23:56:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0717 [2026-10-21 23:57:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1587 [2026-10-21 23:57:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0193 [2026-10-21 23:57:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8513 [2026-10-21 23:57:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9908 [2026-10-21 23:57:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7150 [2026-10-21 23:57:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7056 [2026-10-21 23:57:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7827 [2026-10-21 23:57:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6799 [2026-10-21 23:57:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5750 [2026-10-21 23:57:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9080 [2026-10-21 23:57:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1221 [2026-10-21 23:57:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4361 [2026-10-21 23:57:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8578 [2026-10-21 23:57:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1996 [2026-10-21 23:57:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0391 [2026-10-21 23:57:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9222 [2026-10-21 23:58:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8017 [2026-10-21 23:58:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1667 [2026-10-21 23:58:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0949 [2026-10-21 23:58:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1554 [2026-10-21 23:58:28][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,595 | MFU: 1.8% [2026-10-21 23:58:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8961 [2026-10-21 23:58:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.0516 [2026-10-21 23:58:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6922 [2026-10-21 23:58:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9193 [2026-10-21 23:58:33][INFO][training.pretrain_engine] [Ph2|Step 431] loss=5.0553 | ppl=156.86 | aux=0.02506 | lr=2.98e-04 | gnorm=0.637 | tok/s=29,338 | MFU=7.0% | step=142966ms | tokens=1.254B | mem=4.2/32.7GB | cost=$35.33/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 10:07 (130.1h) | Phase3: SKIPPED | SFT: Oct 27 14:51 (134.9h) | DPO (Final): Oct 27 15:25 (135.5h) [2026-10-21 23:58:33][INFO][training.pretrain_engine] Starting global step 432 (accumulating 32 micro-batches)... [2026-10-21 23:58:33][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-21 23:58:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5468 [2026-10-21 23:58:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2997 [2026-10-21 23:58:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0188 [2026-10-21 23:58:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8403 [2026-10-21 23:59:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2436 [2026-10-21 23:59:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4874 [2026-10-21 23:59:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0203 [2026-10-21 23:59:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4702 [2026-10-21 23:59:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4764 [2026-10-21 23:59:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2511 [2026-10-21 23:59:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9448 [2026-10-21 23:59:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4687 [2026-10-21 23:59:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9032 [2026-10-21 23:59:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5982 [2026-10-21 23:59:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4112 [2026-10-21 23:59:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1518 [2026-10-22 00:00:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7911 [2026-10-22 00:00:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8448 [2026-10-22 00:00:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9325 [2026-10-22 00:00:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8706 [2026-10-22 00:00:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6230 [2026-10-22 00:00:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2993 [2026-10-22 00:00:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0303 [2026-10-22 00:00:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6052 [2026-10-22 00:00:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2667 [2026-10-22 00:00:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1578 [2026-10-22 00:00:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1808 [2026-10-22 00:00:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2300 [2026-10-22 00:00:51][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,593 | MFU: 1.8% [2026-10-22 00:00:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2064 [2026-10-22 00:00:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7468 [2026-10-22 00:00:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0782 [2026-10-22 00:00:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5849 [2026-10-22 00:00:56][INFO][training.pretrain_engine] [Ph2|Step 432] loss=5.0665 | ppl=158.62 | aux=0.02505 | lr=2.98e-04 | gnorm=0.562 | tok/s=29,339 | MFU=7.0% | step=142960ms | tokens=1.255B | mem=4.2/32.7GB | cost=$35.41/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 10:08 (130.1h) | Phase3: SKIPPED | SFT: Oct 27 14:53 (134.9h) | DPO (Final): Oct 27 15:27 (135.4h) [2026-10-22 00:00:59][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step432 | Tokens: 1.255B | Spent: $35.41 | Remaining: $964.59 [2026-10-22 00:00:59][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step432 [2026-10-22 00:00:59][INFO][training.pretrain_engine] Starting global step 433 (accumulating 32 micro-batches)... [2026-10-22 00:00:59][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 00:00:59][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-22 00:00:59][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-22 00:00:59][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-22 00:00:59][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step432 to Phase-Technologies/forge-500m-pretrain (path: phase2_step432)... [2026-10-22 00:00:59][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-22 00:01:08][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-22 00:01:08][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step432 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-22 00:01:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7320 [2026-10-22 00:01:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9218 [2026-10-22 00:01:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8709 [2026-10-22 00:01:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9341 [2026-10-22 00:01:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9160 [2026-10-22 00:01:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9144 [2026-10-22 00:01:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2155 [2026-10-22 00:01:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9515 [2026-10-22 00:01:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1800 [2026-10-22 00:01:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0338 [2026-10-22 00:01:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9635 [2026-10-22 00:01:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0470 [2026-10-22 00:02:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2141 [2026-10-22 00:02:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5164 [2026-10-22 00:02:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4819 [2026-10-22 00:02:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4744 [2026-10-22 00:02:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2532 [2026-10-22 00:02:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1974 [2026-10-22 00:02:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9951 [2026-10-22 00:02:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2782 [2026-10-22 00:02:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1997 [2026-10-22 00:02:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5679 [2026-10-22 00:02:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8748 [2026-10-22 00:02:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0494 [2026-10-22 00:03:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2874 [2026-10-22 00:03:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7248 [2026-10-22 00:03:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1366 [2026-10-22 00:03:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0573 [2026-10-22 00:03:17][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,585 | MFU: 1.8% [2026-10-22 00:03:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2318 [2026-10-22 00:03:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0676 [2026-10-22 00:03:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1560 [2026-10-22 00:03:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4349 [2026-10-22 00:03:22][INFO][training.pretrain_engine] [Ph2|Step 433] loss=4.9788 | ppl=145.29 | aux=0.02505 | lr=2.98e-04 | gnorm=0.582 | tok/s=29,341 | MFU=7.0% | step=142951ms | tokens=1.256B | mem=4.2/32.7GB | cost=$35.50/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 10:10 (130.1h) | Phase3: SKIPPED | SFT: Oct 27 14:54 (134.8h) | DPO (Final): Oct 27 15:28 (135.4h) [2026-10-22 00:03:22][INFO][training.pretrain_engine] Starting global step 434 (accumulating 32 micro-batches)... [2026-10-22 00:03:22][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 00:03:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8546 [2026-10-22 00:03:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1083 [2026-10-22 00:03:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7833 [2026-10-22 00:03:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3661 [2026-10-22 00:03:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4900 [2026-10-22 00:03:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7836 [2026-10-22 00:03:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2791 [2026-10-22 00:03:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1900 [2026-10-22 00:04:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6894 [2026-10-22 00:04:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6980 [2026-10-22 00:04:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9406 [2026-10-22 00:04:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1558 [2026-10-22 00:04:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0311 [2026-10-22 00:04:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6624 [2026-10-22 00:04:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9511 [2026-10-22 00:04:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9690 [2026-10-22 00:04:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5695 [2026-10-22 00:04:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7881 [2026-10-22 00:04:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1647 [2026-10-22 00:04:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5106 [2026-10-22 00:05:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2629 [2026-10-22 00:05:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4862 [2026-10-22 00:05:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8932 [2026-10-22 00:05:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1427 [2026-10-22 00:05:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3049 [2026-10-22 00:05:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2808 [2026-10-22 00:05:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9136 [2026-10-22 00:05:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4995 [2026-10-22 00:05:40][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,588 | MFU: 1.8% [2026-10-22 00:05:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6454 [2026-10-22 00:05:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3606 [2026-10-22 00:05:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9901 [2026-10-22 00:05:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2543 [2026-10-22 00:05:45][INFO][training.pretrain_engine] [Ph2|Step 434] loss=4.9904 | ppl=146.99 | aux=0.02506 | lr=2.98e-04 | gnorm=0.531 | tok/s=29,340 | MFU=7.0% | step=142955ms | tokens=1.257B | mem=4.2/32.7GB | cost=$35.58/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 10:12 (130.1h) | Phase3: SKIPPED | SFT: Oct 27 14:56 (134.8h) | DPO (Final): Oct 27 15:30 (135.4h) [2026-10-22 00:05:45][INFO][training.pretrain_engine] Starting global step 435 (accumulating 32 micro-batches)... [2026-10-22 00:05:45][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 00:06:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8785 [2026-10-22 00:06:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5377 [2026-10-22 00:06:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0833 [2026-10-22 00:06:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8791 [2026-10-22 00:06:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1148 [2026-10-22 00:06:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0843 [2026-10-22 00:06:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1553 [2026-10-22 00:06:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6743 [2026-10-22 00:06:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3671 [2026-10-22 00:06:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5193 [2026-10-22 00:06:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7624 [2026-10-22 00:06:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5807 [2026-10-22 00:06:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0558 [2026-10-22 00:06:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9458 [2026-10-22 00:06:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3728 [2026-10-22 00:06:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7857 [2026-10-22 00:07:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3555 [2026-10-22 00:07:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7953 [2026-10-22 00:07:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0203 [2026-10-22 00:07:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8657 [2026-10-22 00:07:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7245 [2026-10-22 00:07:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2526 [2026-10-22 00:07:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8324 [2026-10-22 00:07:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5104 [2026-10-22 00:07:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5467 [2026-10-22 00:07:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9507 [2026-10-22 00:07:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1418 [2026-10-22 00:07:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3278 [2026-10-22 00:08:03][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,601 | MFU: 1.8% [2026-10-22 00:08:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7878 [2026-10-22 00:08:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2217 [2026-10-22 00:08:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9510 [2026-10-22 00:08:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1250 [2026-10-22 00:08:08][INFO][training.pretrain_engine] [Ph2|Step 435] loss=4.9836 | ppl=146.00 | aux=0.02507 | lr=2.98e-04 | gnorm=0.555 | tok/s=29,342 | MFU=7.0% | step=142947ms | tokens=1.258B | mem=4.2/32.7GB | cost=$35.66/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 10:13 (130.1h) | Phase3: SKIPPED | SFT: Oct 27 14:57 (134.8h) | DPO (Final): Oct 27 15:31 (135.4h) [2026-10-22 00:08:08][INFO][training.pretrain_engine] Starting global step 436 (accumulating 32 micro-batches)... [2026-10-22 00:08:08][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 00:08:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5118 [2026-10-22 00:08:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8079 [2026-10-22 00:08:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5307 [2026-10-22 00:08:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2086 [2026-10-22 00:08:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8781 [2026-10-22 00:08:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2185 [2026-10-22 00:08:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3325 [2026-10-22 00:08:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6075 [2026-10-22 00:09:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1369 [2026-10-22 00:09:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8135 [2026-10-22 00:09:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8009 [2026-10-22 00:09:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8643 [2026-10-22 00:09:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1118 [2026-10-22 00:09:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4486 [2026-10-22 00:09:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8638 [2026-10-22 00:09:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8366 [2026-10-22 00:09:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2792 [2026-10-22 00:09:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1090 [2026-10-22 00:09:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8550 [2026-10-22 00:09:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2639 [2026-10-22 00:09:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9222 [2026-10-22 00:09:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8999 [2026-10-22 00:09:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6973 [2026-10-22 00:09:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7845 [2026-10-22 00:10:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5741 [2026-10-22 00:10:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2636 [2026-10-22 00:10:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3914 [2026-10-22 00:10:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3307 [2026-10-22 00:10:26][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,604 | MFU: 1.8% [2026-10-22 00:10:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7683 [2026-10-22 00:10:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8738 [2026-10-22 00:10:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0632 [2026-10-22 00:10:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8821 [2026-10-22 00:10:31][INFO][training.pretrain_engine] [Ph2|Step 436] loss=4.9725 | ppl=144.38 | aux=0.02506 | lr=2.98e-04 | gnorm=0.516 | tok/s=29,343 | MFU=7.0% | step=142943ms | tokens=1.259B | mem=4.2/32.7GB | cost=$35.74/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 10:15 (130.1h) | Phase3: SKIPPED | SFT: Oct 27 14:59 (134.8h) | DPO (Final): Oct 27 15:33 (135.4h) [2026-10-22 00:10:31][INFO][training.pretrain_engine] Starting global step 437 (accumulating 32 micro-batches)... [2026-10-22 00:10:31][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 00:10:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9532 [2026-10-22 00:10:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2231 [2026-10-22 00:10:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0785 [2026-10-22 00:10:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1890 [2026-10-22 00:11:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9902 [2026-10-22 00:11:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0298 [2026-10-22 00:11:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0985 [2026-10-22 00:11:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7902 [2026-10-22 00:11:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1201 [2026-10-22 00:11:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9733 [2026-10-22 00:11:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2240 [2026-10-22 00:11:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9947 [2026-10-22 00:11:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1965 [2026-10-22 00:11:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7448 [2026-10-22 00:11:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9901 [2026-10-22 00:11:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9360 [2026-10-22 00:12:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.1389 [2026-10-22 00:12:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1442 [2026-10-22 00:12:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6310 [2026-10-22 00:12:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7267 [2026-10-22 00:12:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7033 [2026-10-22 00:12:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6932 [2026-10-22 00:12:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7317 [2026-10-22 00:12:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2113 [2026-10-22 00:12:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1677 [2026-10-22 00:12:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5000 [2026-10-22 00:12:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2088 [2026-10-22 00:12:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5803 [2026-10-22 00:12:49][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,558 | MFU: 1.8% [2026-10-22 00:12:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8154 [2026-10-22 00:12:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7817 [2026-10-22 00:12:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8230 [2026-10-22 00:12:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1041 [2026-10-22 00:12:55][INFO][training.pretrain_engine] [Ph2|Step 437] loss=5.0032 | ppl=148.89 | aux=0.02507 | lr=2.98e-04 | gnorm=0.469 | tok/s=29,341 | MFU=7.0% | step=142951ms | tokens=1.260B | mem=4.2/32.7GB | cost=$35.82/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 10:17 (130.1h) | Phase3: SKIPPED | SFT: Oct 27 15:01 (134.8h) | DPO (Final): Oct 27 15:35 (135.4h) [2026-10-22 00:12:55][INFO][training.pretrain_engine] Starting global step 438 (accumulating 32 micro-batches)... [2026-10-22 00:12:55][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 00:13:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9750 [2026-10-22 00:13:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8927 [2026-10-22 00:13:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9723 [2026-10-22 00:13:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1714 [2026-10-22 00:13:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2639 [2026-10-22 00:13:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5889 [2026-10-22 00:13:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5041 [2026-10-22 00:13:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2064 [2026-10-22 00:13:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5238 [2026-10-22 00:13:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8585 [2026-10-22 00:13:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5491 [2026-10-22 00:13:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7767 [2026-10-22 00:14:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9449 [2026-10-22 00:14:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0958 [2026-10-22 00:14:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3639 [2026-10-22 00:14:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4755 [2026-10-22 00:14:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9938 [2026-10-22 00:14:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2105 [2026-10-22 00:14:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7926 [2026-10-22 00:14:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6344 [2026-10-22 00:14:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9098 [2026-10-22 00:14:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5026 [2026-10-22 00:14:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3318 [2026-10-22 00:14:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0196 [2026-10-22 00:14:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8094 [2026-10-22 00:15:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7355 [2026-10-22 00:15:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0408 [2026-10-22 00:15:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6017 [2026-10-22 00:15:13][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,597 | MFU: 1.8% [2026-10-22 00:15:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2699 [2026-10-22 00:15:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0454 [2026-10-22 00:15:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4538 [2026-10-22 00:15:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6519 [2026-10-22 00:15:18][INFO][training.pretrain_engine] [Ph2|Step 438] loss=4.9680 | ppl=143.74 | aux=0.02505 | lr=2.98e-04 | gnorm=0.482 | tok/s=29,340 | MFU=7.0% | step=142956ms | tokens=1.261B | mem=4.2/32.7GB | cost=$35.91/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 10:19 (130.1h) | Phase3: SKIPPED | SFT: Oct 27 15:03 (134.8h) | DPO (Final): Oct 27 15:37 (135.4h) [2026-10-22 00:15:18][INFO][training.pretrain_engine] Starting global step 439 (accumulating 32 micro-batches)... [2026-10-22 00:15:18][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 00:15:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.9671 [2026-10-22 00:15:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2458 [2026-10-22 00:15:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0503 [2026-10-22 00:15:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5100 [2026-10-22 00:15:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1602 [2026-10-22 00:15:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2248 [2026-10-22 00:15:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4209 [2026-10-22 00:15:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8159 [2026-10-22 00:16:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2325 [2026-10-22 00:16:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3098 [2026-10-22 00:16:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2987 [2026-10-22 00:16:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4793 [2026-10-22 00:16:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9920 [2026-10-22 00:16:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5226 [2026-10-22 00:16:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8738 [2026-10-22 00:16:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4160 [2026-10-22 00:16:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6254 [2026-10-22 00:16:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9661 [2026-10-22 00:16:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0655 [2026-10-22 00:16:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9307 [2026-10-22 00:17:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9439 [2026-10-22 00:17:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8454 [2026-10-22 00:17:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2513 [2026-10-22 00:17:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9198 [2026-10-22 00:17:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2046 [2026-10-22 00:17:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8857 [2026-10-22 00:17:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8094 [2026-10-22 00:17:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6236 [2026-10-22 00:17:36][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,587 | MFU: 1.8% [2026-10-22 00:17:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6670 [2026-10-22 00:17:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0516 [2026-10-22 00:17:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9598 [2026-10-22 00:17:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8382 [2026-10-22 00:17:42][INFO][training.pretrain_engine] [Ph2|Step 439] loss=5.0067 | ppl=149.41 | aux=0.02506 | lr=2.98e-04 | gnorm=0.539 | tok/s=29,339 | MFU=7.0% | step=142961ms | tokens=1.262B | mem=4.2/32.7GB | cost=$35.99/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 10:21 (130.1h) | Phase3: SKIPPED | SFT: Oct 27 15:05 (134.8h) | DPO (Final): Oct 27 15:39 (135.4h) [2026-10-22 00:17:42][INFO][training.pretrain_engine] Starting global step 440 (accumulating 32 micro-batches)... [2026-10-22 00:17:42][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 00:17:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2988 [2026-10-22 00:17:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6252 [2026-10-22 00:17:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4748 [2026-10-22 00:17:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2196 [2026-10-22 00:18:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0646 [2026-10-22 00:18:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1844 [2026-10-22 00:18:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8324 [2026-10-22 00:18:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2220 [2026-10-22 00:18:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9729 [2026-10-22 00:18:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5394 [2026-10-22 00:18:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9101 [2026-10-22 00:18:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3654 [2026-10-22 00:18:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5651 [2026-10-22 00:18:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1236 [2026-10-22 00:18:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1251 [2026-10-22 00:18:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1699 [2026-10-22 00:19:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4952 [2026-10-22 00:19:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5387 [2026-10-22 00:19:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8240 [2026-10-22 00:19:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4575 [2026-10-22 00:19:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5759 [2026-10-22 00:19:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3276 [2026-10-22 00:19:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2167 [2026-10-22 00:19:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5074 [2026-10-22 00:19:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8433 [2026-10-22 00:19:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0326 [2026-10-22 00:19:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6330 [2026-10-22 00:19:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3091 [2026-10-22 00:20:00][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,594 | MFU: 1.8% [2026-10-22 00:20:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1500 [2026-10-22 00:20:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4359 [2026-10-22 00:20:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3781 [2026-10-22 00:20:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0636 [2026-10-22 00:20:05][INFO][training.pretrain_engine] [Ph2|Step 440] loss=5.0337 | ppl=153.51 | aux=0.02505 | lr=2.98e-04 | gnorm=0.664 | tok/s=29,339 | MFU=7.0% | step=142961ms | tokens=1.264B | mem=4.2/32.7GB | cost=$36.07/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 10:23 (130.1h) | Phase3: SKIPPED | SFT: Oct 27 15:07 (134.8h) | DPO (Final): Oct 27 15:41 (135.4h) [2026-10-22 00:20:05][INFO][training.pretrain_engine] Starting global step 441 (accumulating 32 micro-batches)... [2026-10-22 00:20:05][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 00:20:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2241 [2026-10-22 00:20:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4483 [2026-10-22 00:20:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1514 [2026-10-22 00:20:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0828 [2026-10-22 00:20:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5728 [2026-10-22 00:20:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6032 [2026-10-22 00:20:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5289 [2026-10-22 00:20:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3075 [2026-10-22 00:20:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6512 [2026-10-22 00:20:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9584 [2026-10-22 00:20:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7252 [2026-10-22 00:20:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6670 [2026-10-22 00:21:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5498 [2026-10-22 00:21:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4343 [2026-10-22 00:21:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2741 [2026-10-22 00:21:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6722 [2026-10-22 00:21:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4995 [2026-10-22 00:21:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5601 [2026-10-22 00:21:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6686 [2026-10-22 00:21:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8170 [2026-10-22 00:21:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0837 [2026-10-22 00:21:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1543 [2026-10-22 00:21:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0759 [2026-10-22 00:21:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0359 [2026-10-22 00:22:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5174 [2026-10-22 00:22:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1976 [2026-10-22 00:22:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0979 [2026-10-22 00:22:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6369 [2026-10-22 00:22:23][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,601 | MFU: 1.8% [2026-10-22 00:22:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1285 [2026-10-22 00:22:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2460 [2026-10-22 00:22:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5009 [2026-10-22 00:22:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6554 [2026-10-22 00:22:28][INFO][training.pretrain_engine] [Ph2|Step 441] loss=5.0014 | ppl=148.61 | aux=0.02507 | lr=2.98e-04 | gnorm=0.832 | tok/s=29,340 | MFU=7.0% | step=142956ms | tokens=1.265B | mem=4.2/32.7GB | cost=$36.15/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 10:24 (130.0h) | Phase3: SKIPPED | SFT: Oct 27 15:08 (134.8h) | DPO (Final): Oct 27 15:43 (135.3h) [2026-10-22 00:22:31][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step441 | Tokens: 1.265B | Spent: $36.15 | Remaining: $963.85 [2026-10-22 00:22:31][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step441 [2026-10-22 00:22:31][INFO][training.pretrain_engine] Starting global step 442 (accumulating 32 micro-batches)... [2026-10-22 00:22:31][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 00:22:31][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-22 00:22:31][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-22 00:22:31][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-22 00:22:31][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step441 to Phase-Technologies/forge-500m-pretrain (path: phase2_step441)... [2026-10-22 00:22:32][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-22 00:22:41][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-22 00:22:41][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step441 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-22 00:22:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6277 [2026-10-22 00:22:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1529 [2026-10-22 00:22:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3454 [2026-10-22 00:22:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6458 [2026-10-22 00:23:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3077 [2026-10-22 00:23:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9809 [2026-10-22 00:23:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9160 [2026-10-22 00:23:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0176 [2026-10-22 00:23:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2403 [2026-10-22 00:23:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8790 [2026-10-22 00:23:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6747 [2026-10-22 00:23:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0487 [2026-10-22 00:23:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0936 [2026-10-22 00:23:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1448 [2026-10-22 00:23:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0719 [2026-10-22 00:23:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0717 [2026-10-22 00:23:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3938 [2026-10-22 00:23:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1136 [2026-10-22 00:23:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9411 [2026-10-22 00:24:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2128 [2026-10-22 00:24:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9738 [2026-10-22 00:24:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3830 [2026-10-22 00:24:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8454 [2026-10-22 00:24:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7845 [2026-10-22 00:24:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4625 [2026-10-22 00:24:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3007 [2026-10-22 00:24:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8945 [2026-10-22 00:24:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4815 [2026-10-22 00:24:49][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,605 | MFU: 1.8% [2026-10-22 00:24:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9127 [2026-10-22 00:24:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1963 [2026-10-22 00:24:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1287 [2026-10-22 00:24:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1486 [2026-10-22 00:24:54][INFO][training.pretrain_engine] [Ph2|Step 442] loss=4.9886 | ppl=146.74 | aux=0.02506 | lr=2.98e-04 | gnorm=0.961 | tok/s=29,341 | MFU=7.0% | step=142950ms | tokens=1.266B | mem=4.2/32.7GB | cost=$36.23/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 10:26 (130.0h) | Phase3: SKIPPED | SFT: Oct 27 15:10 (134.8h) | DPO (Final): Oct 27 15:44 (135.3h) [2026-10-22 00:24:54][INFO][training.pretrain_engine] Starting global step 443 (accumulating 32 micro-batches)... [2026-10-22 00:24:54][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 00:25:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9686 [2026-10-22 00:25:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8923 [2026-10-22 00:25:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2389 [2026-10-22 00:25:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9327 [2026-10-22 00:25:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3832 [2026-10-22 00:25:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9483 [2026-10-22 00:25:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9292 [2026-10-22 00:25:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9524 [2026-10-22 00:25:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1903 [2026-10-22 00:25:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2372 [2026-10-22 00:25:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7457 [2026-10-22 00:25:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0746 [2026-10-22 00:26:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=3.5776 [2026-10-22 00:26:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=3.8522 [2026-10-22 00:26:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.1659 [2026-10-22 00:26:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=3.7733 [2026-10-22 00:26:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2924 [2026-10-22 00:26:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5757 [2026-10-22 00:26:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0604 [2026-10-22 00:26:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9127 [2026-10-22 00:26:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3068 [2026-10-22 00:26:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3037 [2026-10-22 00:26:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9475 [2026-10-22 00:26:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4568 [2026-10-22 00:26:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1712 [2026-10-22 00:26:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1747 [2026-10-22 00:26:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8740 [2026-10-22 00:26:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8856 [2026-10-22 00:27:12][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,592 | MFU: 1.8% [2026-10-22 00:27:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2054 [2026-10-22 00:27:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3248 [2026-10-22 00:27:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2286 [2026-10-22 00:27:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4824 [2026-10-22 00:27:17][INFO][training.pretrain_engine] [Ph2|Step 443] loss=4.9742 | ppl=144.63 | aux=0.02507 | lr=2.98e-04 | gnorm=0.738 | tok/s=29,339 | MFU=7.0% | step=142962ms | tokens=1.267B | mem=4.2/32.7GB | cost=$36.32/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 10:28 (130.0h) | Phase3: SKIPPED | SFT: Oct 27 15:12 (134.8h) | DPO (Final): Oct 27 15:47 (135.3h) [2026-10-22 00:27:17][INFO][training.pretrain_engine] Starting global step 444 (accumulating 32 micro-batches)... [2026-10-22 00:27:17][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 00:27:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0002 [2026-10-22 00:27:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0892 [2026-10-22 00:27:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5378 [2026-10-22 00:27:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4776 [2026-10-22 00:27:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1667 [2026-10-22 00:27:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1003 [2026-10-22 00:27:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0411 [2026-10-22 00:27:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5815 [2026-10-22 00:28:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4598 [2026-10-22 00:28:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6111 [2026-10-22 00:28:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6391 [2026-10-22 00:28:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3729 [2026-10-22 00:28:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5866 [2026-10-22 00:28:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5837 [2026-10-22 00:28:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9393 [2026-10-22 00:28:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6133 [2026-10-22 00:28:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9290 [2026-10-22 00:28:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4622 [2026-10-22 00:28:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1723 [2026-10-22 00:28:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1037 [2026-10-22 00:29:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3115 [2026-10-22 00:29:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8524 [2026-10-22 00:29:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3590 [2026-10-22 00:29:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2377 [2026-10-22 00:29:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0911 [2026-10-22 00:29:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9742 [2026-10-22 00:29:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3127 [2026-10-22 00:29:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3620 [2026-10-22 00:29:35][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,603 | MFU: 1.8% [2026-10-22 00:29:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0140 [2026-10-22 00:29:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4818 [2026-10-22 00:29:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3307 [2026-10-22 00:29:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8230 [2026-10-22 00:29:40][INFO][training.pretrain_engine] [Ph2|Step 444] loss=5.1027 | ppl=164.46 | aux=0.02507 | lr=2.98e-04 | gnorm=0.699 | tok/s=29,339 | MFU=7.0% | step=142960ms | tokens=1.268B | mem=4.2/32.7GB | cost=$36.40/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 10:30 (130.0h) | Phase3: SKIPPED | SFT: Oct 27 15:14 (134.7h) | DPO (Final): Oct 27 15:48 (135.3h) [2026-10-22 00:29:40][INFO][training.pretrain_engine] Starting global step 445 (accumulating 32 micro-batches)... [2026-10-22 00:29:40][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 00:29:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6431 [2026-10-22 00:29:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5357 [2026-10-22 00:29:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4675 [2026-10-22 00:29:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8630 [2026-10-22 00:30:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7930 [2026-10-22 00:30:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.5312 [2026-10-22 00:30:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7123 [2026-10-22 00:30:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4539 [2026-10-22 00:30:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.1188 [2026-10-22 00:30:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8033 [2026-10-22 00:30:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9472 [2026-10-22 00:30:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1185 [2026-10-22 00:30:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2126 [2026-10-22 00:30:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4305 [2026-10-22 00:30:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4968 [2026-10-22 00:30:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5083 [2026-10-22 00:31:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1056 [2026-10-22 00:31:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4938 [2026-10-22 00:31:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1391 [2026-10-22 00:31:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6208 [2026-10-22 00:31:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0766 [2026-10-22 00:31:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8544 [2026-10-22 00:31:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7330 [2026-10-22 00:31:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8737 [2026-10-22 00:31:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3593 [2026-10-22 00:31:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3853 [2026-10-22 00:31:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5828 [2026-10-22 00:31:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6869 [2026-10-22 00:31:59][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,572 | MFU: 1.8% [2026-10-22 00:32:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9201 [2026-10-22 00:32:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8444 [2026-10-22 00:32:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9008 [2026-10-22 00:32:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6739 [2026-10-22 00:32:04][INFO][training.pretrain_engine] [Ph2|Step 445] loss=4.9678 | ppl=143.71 | aux=0.02506 | lr=2.98e-04 | gnorm=0.637 | tok/s=29,339 | MFU=7.0% | step=142962ms | tokens=1.269B | mem=4.2/32.7GB | cost=$36.48/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 10:32 (130.0h) | Phase3: SKIPPED | SFT: Oct 27 15:16 (134.7h) | DPO (Final): Oct 27 15:50 (135.3h) [2026-10-22 00:32:04][INFO][training.pretrain_engine] Starting global step 446 (accumulating 32 micro-batches)... [2026-10-22 00:32:04][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 00:32:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9482 [2026-10-22 00:32:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0445 [2026-10-22 00:32:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9892 [2026-10-22 00:32:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8602 [2026-10-22 00:32:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8958 [2026-10-22 00:32:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1041 [2026-10-22 00:32:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6684 [2026-10-22 00:32:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6431 [2026-10-22 00:32:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1420 [2026-10-22 00:32:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9838 [2026-10-22 00:32:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9879 [2026-10-22 00:32:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9283 [2026-10-22 00:33:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5000 [2026-10-22 00:33:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0640 [2026-10-22 00:33:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3053 [2026-10-22 00:33:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6705 [2026-10-22 00:33:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9117 [2026-10-22 00:33:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4149 [2026-10-22 00:33:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3562 [2026-10-22 00:33:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2924 [2026-10-22 00:33:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8089 [2026-10-22 00:33:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2403 [2026-10-22 00:33:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0656 [2026-10-22 00:33:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9924 [2026-10-22 00:34:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0899 [2026-10-22 00:34:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7445 [2026-10-22 00:34:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7071 [2026-10-22 00:34:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7844 [2026-10-22 00:34:22][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,598 | MFU: 1.8% [2026-10-22 00:34:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7852 [2026-10-22 00:34:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0233 [2026-10-22 00:34:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7838 [2026-10-22 00:34:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6178 [2026-10-22 00:34:28][INFO][training.pretrain_engine] [Ph2|Step 446] loss=4.8785 | ppl=131.43 | aux=0.02506 | lr=2.98e-04 | gnorm=0.586 | tok/s=29,337 | MFU=7.0% | step=142972ms | tokens=1.270B | mem=4.2/32.7GB | cost=$36.56/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 10:34 (130.0h) | Phase3: SKIPPED | SFT: Oct 27 15:18 (134.7h) | DPO (Final): Oct 27 15:52 (135.3h) [2026-10-22 00:34:28][INFO][training.pretrain_engine] Starting global step 447 (accumulating 32 micro-batches)... [2026-10-22 00:34:28][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 00:34:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0537 [2026-10-22 00:34:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1695 [2026-10-22 00:34:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4244 [2026-10-22 00:34:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8550 [2026-10-22 00:35:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1699 [2026-10-22 00:35:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0287 [2026-10-22 00:35:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0756 [2026-10-22 00:35:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9162 [2026-10-22 00:35:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2373 [2026-10-22 00:35:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5911 [2026-10-22 00:35:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1079 [2026-10-22 00:35:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7733 [2026-10-22 00:35:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4957 [2026-10-22 00:35:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7836 [2026-10-22 00:35:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1709 [2026-10-22 00:35:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3425 [2026-10-22 00:35:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8780 [2026-10-22 00:35:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2542 [2026-10-22 00:35:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0123 [2026-10-22 00:35:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5326 [2026-10-22 00:36:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9527 [2026-10-22 00:36:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0219 [2026-10-22 00:36:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8120 [2026-10-22 00:36:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9484 [2026-10-22 00:36:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9328 [2026-10-22 00:36:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8959 [2026-10-22 00:36:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8557 [2026-10-22 00:36:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8452 [2026-10-22 00:36:45][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,612 | MFU: 1.8% [2026-10-22 00:36:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7320 [2026-10-22 00:36:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0981 [2026-10-22 00:36:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1226 [2026-10-22 00:36:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8667 [2026-10-22 00:36:51][INFO][training.pretrain_engine] [Ph2|Step 447] loss=5.0799 | ppl=160.76 | aux=0.02505 | lr=2.98e-04 | gnorm=0.621 | tok/s=29,336 | MFU=7.0% | step=142976ms | tokens=1.271B | mem=4.2/32.7GB | cost=$36.65/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 10:36 (130.0h) | Phase3: SKIPPED | SFT: Oct 27 15:20 (134.7h) | DPO (Final): Oct 27 15:55 (135.3h) [2026-10-22 00:36:51][INFO][training.pretrain_engine] Starting global step 448 (accumulating 32 micro-batches)... [2026-10-22 00:36:51][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 00:37:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5836 [2026-10-22 00:37:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1486 [2026-10-22 00:37:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2447 [2026-10-22 00:37:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7615 [2026-10-22 00:37:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4004 [2026-10-22 00:37:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1890 [2026-10-22 00:37:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8276 [2026-10-22 00:37:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5805 [2026-10-22 00:37:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9567 [2026-10-22 00:37:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9160 [2026-10-22 00:37:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1577 [2026-10-22 00:37:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1341 [2026-10-22 00:38:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5700 [2026-10-22 00:38:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9593 [2026-10-22 00:38:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4844 [2026-10-22 00:38:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5469 [2026-10-22 00:38:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6962 [2026-10-22 00:38:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3497 [2026-10-22 00:38:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0796 [2026-10-22 00:38:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.1045 [2026-10-22 00:38:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5662 [2026-10-22 00:38:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8516 [2026-10-22 00:38:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4053 [2026-10-22 00:38:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6555 [2026-10-22 00:38:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7274 [2026-10-22 00:38:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2897 [2026-10-22 00:38:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5719 [2026-10-22 00:38:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6321 [2026-10-22 00:39:09][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,587 | MFU: 1.8% [2026-10-22 00:39:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6604 [2026-10-22 00:39:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9993 [2026-10-22 00:39:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4931 [2026-10-22 00:39:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8886 [2026-10-22 00:39:15][INFO][training.pretrain_engine] [Ph2|Step 448] loss=4.9315 | ppl=138.59 | aux=0.02506 | lr=2.98e-04 | gnorm=0.605 | tok/s=29,334 | MFU=7.0% | step=142983ms | tokens=1.272B | mem=4.2/32.7GB | cost=$36.73/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 10:39 (130.0h) | Phase3: SKIPPED | SFT: Oct 27 15:23 (134.7h) | DPO (Final): Oct 27 15:57 (135.3h) [2026-10-22 00:39:15][INFO][training.pretrain_engine] Starting global step 449 (accumulating 32 micro-batches)... [2026-10-22 00:39:15][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 00:39:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0230 [2026-10-22 00:39:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9872 [2026-10-22 00:39:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9904 [2026-10-22 00:39:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2824 [2026-10-22 00:39:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8920 [2026-10-22 00:39:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6504 [2026-10-22 00:39:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7980 [2026-10-22 00:39:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3366 [2026-10-22 00:40:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2238 [2026-10-22 00:40:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8942 [2026-10-22 00:40:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8358 [2026-10-22 00:40:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5041 [2026-10-22 00:40:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4745 [2026-10-22 00:40:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1142 [2026-10-22 00:40:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1023 [2026-10-22 00:40:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7175 [2026-10-22 00:40:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1224 [2026-10-22 00:40:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9045 [2026-10-22 00:40:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8057 [2026-10-22 00:40:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6224 [2026-10-22 00:41:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8081 [2026-10-22 00:41:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8787 [2026-10-22 00:41:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7476 [2026-10-22 00:41:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3922 [2026-10-22 00:41:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7096 [2026-10-22 00:41:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5913 [2026-10-22 00:41:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3956 [2026-10-22 00:41:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0509 [2026-10-22 00:41:34][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,573 | MFU: 1.8% [2026-10-22 00:41:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2030 [2026-10-22 00:41:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9081 [2026-10-22 00:41:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5232 [2026-10-22 00:41:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4081 [2026-10-22 00:41:39][INFO][training.pretrain_engine] [Ph2|Step 449] loss=5.0420 | ppl=154.78 | aux=0.02507 | lr=2.98e-04 | gnorm=0.516 | tok/s=29,332 | MFU=7.0% | step=142992ms | tokens=1.273B | mem=4.2/32.7GB | cost=$36.81/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 10:41 (130.0h) | Phase3: SKIPPED | SFT: Oct 27 15:25 (134.7h) | DPO (Final): Oct 27 15:59 (135.3h) [2026-10-22 00:41:39][INFO][training.pretrain_engine] Starting global step 450 (accumulating 32 micro-batches)... [2026-10-22 00:41:39][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 00:41:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2593 [2026-10-22 00:41:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2449 [2026-10-22 00:41:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9962 [2026-10-22 00:41:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0099 [2026-10-22 00:42:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8881 [2026-10-22 00:42:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8900 [2026-10-22 00:42:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4138 [2026-10-22 00:42:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0494 [2026-10-22 00:42:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8475 [2026-10-22 00:42:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6049 [2026-10-22 00:42:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9769 [2026-10-22 00:42:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.1503 [2026-10-22 00:42:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0279 [2026-10-22 00:42:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5476 [2026-10-22 00:42:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3969 [2026-10-22 00:42:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2121 [2026-10-22 00:43:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2348 [2026-10-22 00:43:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6687 [2026-10-22 00:43:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2694 [2026-10-22 00:43:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5611 [2026-10-22 00:43:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7246 [2026-10-22 00:43:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0819 [2026-10-22 00:43:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4099 [2026-10-22 00:43:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6755 [2026-10-22 00:43:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8466 [2026-10-22 00:43:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8759 [2026-10-22 00:43:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7119 [2026-10-22 00:43:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8857 [2026-10-22 00:43:57][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,589 | MFU: 1.8% [2026-10-22 00:44:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9743 [2026-10-22 00:44:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0801 [2026-10-22 00:44:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8466 [2026-10-22 00:44:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7001 [2026-10-22 00:44:03][INFO][training.pretrain_engine] [Ph2|Step 450] loss=5.0229 | ppl=151.85 | aux=0.02505 | lr=2.98e-04 | gnorm=0.629 | tok/s=29,330 | MFU=7.0% | step=143005ms | tokens=1.274B | mem=4.2/32.7GB | cost=$36.89/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 10:43 (130.0h) | Phase3: SKIPPED | SFT: Oct 27 15:27 (134.7h) | DPO (Final): Oct 27 16:02 (135.3h) [2026-10-22 00:44:06][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step450 | Tokens: 1.274B | Spent: $36.89 | Remaining: $963.11 [2026-10-22 00:44:06][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step450 [2026-10-22 00:44:06][INFO][training.pretrain_engine] Starting global step 451 (accumulating 32 micro-batches)... [2026-10-22 00:44:06][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 00:44:06][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-22 00:44:06][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-22 00:44:06][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-22 00:44:06][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step450 to Phase-Technologies/forge-500m-pretrain (path: phase2_step450)... [2026-10-22 00:44:06][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-22 00:44:16][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-22 00:44:16][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step450 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-22 00:44:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4723 [2026-10-22 00:44:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4623 [2026-10-22 00:44:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1467 [2026-10-22 00:44:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3111 [2026-10-22 00:44:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7971 [2026-10-22 00:44:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9170 [2026-10-22 00:44:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4562 [2026-10-22 00:44:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2294 [2026-10-22 00:44:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9345 [2026-10-22 00:44:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0858 [2026-10-22 00:44:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5492 [2026-10-22 00:45:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1619 [2026-10-22 00:45:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6741 [2026-10-22 00:45:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9199 [2026-10-22 00:45:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1989 [2026-10-22 00:45:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1060 [2026-10-22 00:45:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2617 [2026-10-22 00:45:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2852 [2026-10-22 00:45:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7507 [2026-10-22 00:45:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0555 [2026-10-22 00:45:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8652 [2026-10-22 00:45:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7373 [2026-10-22 00:45:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3113 [2026-10-22 00:45:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5553 [2026-10-22 00:46:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0249 [2026-10-22 00:46:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9586 [2026-10-22 00:46:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3423 [2026-10-22 00:46:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8067 [2026-10-22 00:46:25][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,572 | MFU: 1.8% [2026-10-22 00:46:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9533 [2026-10-22 00:46:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7127 [2026-10-22 00:46:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3148 [2026-10-22 00:46:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3248 [2026-10-22 00:46:29][INFO][training.pretrain_engine] [Ph2|Step 451] loss=4.9768 | ppl=145.01 | aux=0.02505 | lr=2.98e-04 | gnorm=0.531 | tok/s=29,329 | MFU=7.0% | step=143008ms | tokens=1.275B | mem=4.2/32.7GB | cost=$36.98/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 10:45 (130.0h) | Phase3: SKIPPED | SFT: Oct 27 15:29 (134.7h) | DPO (Final): Oct 27 16:04 (135.3h) [2026-10-22 00:46:29][INFO][training.pretrain_engine] Starting global step 452 (accumulating 32 micro-batches)... [2026-10-22 00:46:29][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 00:46:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1599 [2026-10-22 00:46:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0854 [2026-10-22 00:46:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0594 [2026-10-22 00:46:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7261 [2026-10-22 00:47:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5849 [2026-10-22 00:47:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0054 [2026-10-22 00:47:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2602 [2026-10-22 00:47:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3289 [2026-10-22 00:47:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6980 [2026-10-22 00:47:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6929 [2026-10-22 00:47:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8667 [2026-10-22 00:47:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6894 [2026-10-22 00:47:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7135 [2026-10-22 00:47:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5254 [2026-10-22 00:47:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9187 [2026-10-22 00:47:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8261 [2026-10-22 00:47:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2738 [2026-10-22 00:47:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1968 [2026-10-22 00:47:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2237 [2026-10-22 00:47:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2428 [2026-10-22 00:48:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5868 [2026-10-22 00:48:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3014 [2026-10-22 00:48:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7890 [2026-10-22 00:48:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5705 [2026-10-22 00:48:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1712 [2026-10-22 00:48:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4138 [2026-10-22 00:48:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1699 [2026-10-22 00:48:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0561 [2026-10-22 00:48:48][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,578 | MFU: 1.8% [2026-10-22 00:48:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5822 [2026-10-22 00:48:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1589 [2026-10-22 00:48:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7857 [2026-10-22 00:48:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5665 [2026-10-22 00:48:53][INFO][training.pretrain_engine] [Ph2|Step 452] loss=4.9162 | ppl=136.49 | aux=0.02506 | lr=2.98e-04 | gnorm=0.605 | tok/s=29,327 | MFU=7.0% | step=143017ms | tokens=1.276B | mem=4.2/32.7GB | cost=$37.06/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 10:48 (130.0h) | Phase3: SKIPPED | SFT: Oct 27 15:32 (134.7h) | DPO (Final): Oct 27 16:06 (135.3h) [2026-10-22 00:48:53][INFO][training.pretrain_engine] Starting global step 453 (accumulating 32 micro-batches)... [2026-10-22 00:48:53][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 00:49:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8358 [2026-10-22 00:49:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6030 [2026-10-22 00:49:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0215 [2026-10-22 00:49:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9152 [2026-10-22 00:49:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0567 [2026-10-22 00:49:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3251 [2026-10-22 00:49:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6736 [2026-10-22 00:49:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5605 [2026-10-22 00:49:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6455 [2026-10-22 00:49:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1001 [2026-10-22 00:49:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8676 [2026-10-22 00:49:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8947 [2026-10-22 00:50:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8697 [2026-10-22 00:50:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2531 [2026-10-22 00:50:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9372 [2026-10-22 00:50:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3654 [2026-10-22 00:50:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4954 [2026-10-22 00:50:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2893 [2026-10-22 00:50:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4531 [2026-10-22 00:50:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3158 [2026-10-22 00:50:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8431 [2026-10-22 00:50:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0006 [2026-10-22 00:50:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9489 [2026-10-22 00:50:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1338 [2026-10-22 00:50:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1136 [2026-10-22 00:50:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7395 [2026-10-22 00:50:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6115 [2026-10-22 00:50:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8572 [2026-10-22 00:51:11][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,593 | MFU: 1.8% [2026-10-22 00:51:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8896 [2026-10-22 00:51:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9057 [2026-10-22 00:51:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8598 [2026-10-22 00:51:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0252 [2026-10-22 00:51:16][INFO][training.pretrain_engine] [Ph2|Step 453] loss=4.9215 | ppl=137.21 | aux=0.02506 | lr=2.98e-04 | gnorm=0.566 | tok/s=29,326 | MFU=7.0% | step=143023ms | tokens=1.277B | mem=4.2/32.7GB | cost=$37.14/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 10:50 (130.0h) | Phase3: SKIPPED | SFT: Oct 27 15:34 (134.7h) | DPO (Final): Oct 27 16:08 (135.3h) [2026-10-22 00:51:16][INFO][training.pretrain_engine] Starting global step 454 (accumulating 32 micro-batches)... [2026-10-22 00:51:16][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 00:51:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6082 [2026-10-22 00:51:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0439 [2026-10-22 00:51:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3370 [2026-10-22 00:51:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5234 [2026-10-22 00:51:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1058 [2026-10-22 00:51:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9452 [2026-10-22 00:51:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7229 [2026-10-22 00:51:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3058 [2026-10-22 00:52:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9281 [2026-10-22 00:52:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6988 [2026-10-22 00:52:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6601 [2026-10-22 00:52:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0943 [2026-10-22 00:52:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6468 [2026-10-22 00:52:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=3.8731 [2026-10-22 00:52:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6254 [2026-10-22 00:52:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5687 [2026-10-22 00:52:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0350 [2026-10-22 00:52:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8666 [2026-10-22 00:52:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0850 [2026-10-22 00:52:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6009 [2026-10-22 00:53:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5506 [2026-10-22 00:53:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1243 [2026-10-22 00:53:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2939 [2026-10-22 00:53:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0019 [2026-10-22 00:53:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1612 [2026-10-22 00:53:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8564 [2026-10-22 00:53:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1420 [2026-10-22 00:53:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8802 [2026-10-22 00:53:34][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,577 | MFU: 1.8% [2026-10-22 00:53:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9473 [2026-10-22 00:53:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9149 [2026-10-22 00:53:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2041 [2026-10-22 00:53:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1049 [2026-10-22 00:53:40][INFO][training.pretrain_engine] [Ph2|Step 454] loss=4.8574 | ppl=128.69 | aux=0.02507 | lr=2.98e-04 | gnorm=0.652 | tok/s=29,324 | MFU=7.0% | step=143034ms | tokens=1.278B | mem=4.2/32.7GB | cost=$37.22/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 10:52 (130.0h) | Phase3: SKIPPED | SFT: Oct 27 15:36 (134.7h) | DPO (Final): Oct 27 16:10 (135.3h) [2026-10-22 00:53:40][INFO][training.pretrain_engine] Starting global step 455 (accumulating 32 micro-batches)... [2026-10-22 00:53:40][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 00:53:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.1584 [2026-10-22 00:53:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5354 [2026-10-22 00:53:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7955 [2026-10-22 00:53:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7030 [2026-10-22 00:54:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6133 [2026-10-22 00:54:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9210 [2026-10-22 00:54:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4444 [2026-10-22 00:54:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7466 [2026-10-22 00:54:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9632 [2026-10-22 00:54:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6418 [2026-10-22 00:54:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0010 [2026-10-22 00:54:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9579 [2026-10-22 00:54:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5027 [2026-10-22 00:54:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6565 [2026-10-22 00:54:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6211 [2026-10-22 00:54:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5124 [2026-10-22 00:55:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2266 [2026-10-22 00:55:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1088 [2026-10-22 00:55:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3808 [2026-10-22 00:55:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2531 [2026-10-22 00:55:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9824 [2026-10-22 00:55:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9356 [2026-10-22 00:55:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0541 [2026-10-22 00:55:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8844 [2026-10-22 00:55:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8018 [2026-10-22 00:55:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9632 [2026-10-22 00:55:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0864 [2026-10-22 00:55:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0987 [2026-10-22 00:55:58][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,586 | MFU: 1.8% [2026-10-22 00:56:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5159 [2026-10-22 00:56:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9561 [2026-10-22 00:56:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5595 [2026-10-22 00:56:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9783 [2026-10-22 00:56:03][INFO][training.pretrain_engine] [Ph2|Step 455] loss=4.9229 | ppl=137.40 | aux=0.02506 | lr=2.98e-04 | gnorm=0.602 | tok/s=29,323 | MFU=7.0% | step=143039ms | tokens=1.279B | mem=4.2/32.7GB | cost=$37.30/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 10:54 (130.0h) | Phase3: SKIPPED | SFT: Oct 27 15:38 (134.7h) | DPO (Final): Oct 27 16:13 (135.3h) [2026-10-22 00:56:03][INFO][training.pretrain_engine] Starting global step 456 (accumulating 32 micro-batches)... [2026-10-22 00:56:03][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 00:56:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0825 [2026-10-22 00:56:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1802 [2026-10-22 00:56:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1645 [2026-10-22 00:56:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9735 [2026-10-22 00:56:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0068 [2026-10-22 00:56:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1247 [2026-10-22 00:56:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1685 [2026-10-22 00:56:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0057 [2026-10-22 00:56:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7092 [2026-10-22 00:56:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=3.9883 [2026-10-22 00:56:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6561 [2026-10-22 00:56:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8847 [2026-10-22 00:57:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7326 [2026-10-22 00:57:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0440 [2026-10-22 00:57:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2953 [2026-10-22 00:57:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0628 [2026-10-22 00:57:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9003 [2026-10-22 00:57:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7992 [2026-10-22 00:57:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7907 [2026-10-22 00:57:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0208 [2026-10-22 00:57:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8085 [2026-10-22 00:57:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0976 [2026-10-22 00:57:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8225 [2026-10-22 00:57:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6682 [2026-10-22 00:58:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8922 [2026-10-22 00:58:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8124 [2026-10-22 00:58:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8221 [2026-10-22 00:58:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6371 [2026-10-22 00:58:21][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,584 | MFU: 1.8% [2026-10-22 00:58:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8837 [2026-10-22 00:58:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1981 [2026-10-22 00:58:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0520 [2026-10-22 00:58:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8835 [2026-10-22 00:58:26][INFO][training.pretrain_engine] [Ph2|Step 456] loss=4.9581 | ppl=142.32 | aux=0.02506 | lr=2.98e-04 | gnorm=0.637 | tok/s=29,322 | MFU=7.0% | step=143041ms | tokens=1.280B | mem=4.2/32.7GB | cost=$37.39/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 10:56 (130.0h) | Phase3: SKIPPED | SFT: Oct 27 15:40 (134.7h) | DPO (Final): Oct 27 16:14 (135.3h) [2026-10-22 00:58:26][INFO][training.pretrain_engine] Starting global step 457 (accumulating 32 micro-batches)... [2026-10-22 00:58:26][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 00:58:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2508 [2026-10-22 00:58:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0092 [2026-10-22 00:58:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9769 [2026-10-22 00:58:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5521 [2026-10-22 00:59:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1549 [2026-10-22 00:59:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0000 [2026-10-22 00:59:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1190 [2026-10-22 00:59:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6741 [2026-10-22 00:59:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3429 [2026-10-22 00:59:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9064 [2026-10-22 00:59:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1023 [2026-10-22 00:59:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0127 [2026-10-22 00:59:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1524 [2026-10-22 00:59:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2990 [2026-10-22 00:59:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9833 [2026-10-22 00:59:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1532 [2026-10-22 00:59:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6770 [2026-10-22 00:59:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8319 [2026-10-22 00:59:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5292 [2026-10-22 00:59:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6170 [2026-10-22 01:00:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8101 [2026-10-22 01:00:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4353 [2026-10-22 01:00:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7564 [2026-10-22 01:00:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.1894 [2026-10-22 01:00:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4747 [2026-10-22 01:00:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4113 [2026-10-22 01:00:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3778 [2026-10-22 01:00:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1890 [2026-10-22 01:00:44][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,596 | MFU: 1.8% [2026-10-22 01:00:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6435 [2026-10-22 01:00:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2098 [2026-10-22 01:00:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1222 [2026-10-22 01:00:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8119 [2026-10-22 01:00:49][INFO][training.pretrain_engine] [Ph2|Step 457] loss=4.9443 | ppl=140.37 | aux=0.02506 | lr=2.98e-04 | gnorm=0.617 | tok/s=29,321 | MFU=7.0% | step=143047ms | tokens=1.281B | mem=4.2/32.7GB | cost=$37.47/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 10:58 (130.0h) | Phase3: SKIPPED | SFT: Oct 27 15:42 (134.7h) | DPO (Final): Oct 27 16:17 (135.3h) [2026-10-22 01:00:49][INFO][training.pretrain_engine] Starting global step 458 (accumulating 32 micro-batches)... [2026-10-22 01:00:49][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 01:01:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7614 [2026-10-22 01:01:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1564 [2026-10-22 01:01:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8661 [2026-10-22 01:01:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8434 [2026-10-22 01:01:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3022 [2026-10-22 01:01:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0296 [2026-10-22 01:01:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9183 [2026-10-22 01:01:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9553 [2026-10-22 01:01:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1211 [2026-10-22 01:01:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0367 [2026-10-22 01:01:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4546 [2026-10-22 01:01:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2017 [2026-10-22 01:02:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2667 [2026-10-22 01:02:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8459 [2026-10-22 01:02:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1310 [2026-10-22 01:02:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1164 [2026-10-22 01:02:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3378 [2026-10-22 01:02:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3678 [2026-10-22 01:02:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4010 [2026-10-22 01:02:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4069 [2026-10-22 01:02:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7082 [2026-10-22 01:02:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5249 [2026-10-22 01:02:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6494 [2026-10-22 01:02:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7112 [2026-10-22 01:02:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4332 [2026-10-22 01:02:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1374 [2026-10-22 01:02:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9521 [2026-10-22 01:02:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0725 [2026-10-22 01:03:07][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,582 | MFU: 1.8% [2026-10-22 01:03:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8930 [2026-10-22 01:03:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0639 [2026-10-22 01:03:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9888 [2026-10-22 01:03:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9295 [2026-10-22 01:03:12][INFO][training.pretrain_engine] [Ph2|Step 458] loss=5.0343 | ppl=153.59 | aux=0.02506 | lr=2.98e-04 | gnorm=0.531 | tok/s=29,319 | MFU=7.0% | step=143057ms | tokens=1.282B | mem=4.2/32.7GB | cost=$37.55/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 11:01 (130.0h) | Phase3: SKIPPED | SFT: Oct 27 15:45 (134.7h) | DPO (Final): Oct 27 16:19 (135.3h) [2026-10-22 01:03:12][INFO][training.pretrain_engine] Starting global step 459 (accumulating 32 micro-batches)... [2026-10-22 01:03:12][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 01:03:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2258 [2026-10-22 01:03:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9284 [2026-10-22 01:03:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0771 [2026-10-22 01:03:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1636 [2026-10-22 01:03:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6031 [2026-10-22 01:03:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4998 [2026-10-22 01:03:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9880 [2026-10-22 01:03:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3256 [2026-10-22 01:04:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2694 [2026-10-22 01:04:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2345 [2026-10-22 01:04:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9512 [2026-10-22 01:04:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2148 [2026-10-22 01:04:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6431 [2026-10-22 01:04:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6962 [2026-10-22 01:04:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4868 [2026-10-22 01:04:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=3.6725 [2026-10-22 01:04:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7617 [2026-10-22 01:04:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8489 [2026-10-22 01:04:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9714 [2026-10-22 01:04:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6861 [2026-10-22 01:04:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.1722 [2026-10-22 01:04:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=3.9359 [2026-10-22 01:04:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4685 [2026-10-22 01:05:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5642 [2026-10-22 01:05:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9310 [2026-10-22 01:05:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5180 [2026-10-22 01:05:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6730 [2026-10-22 01:05:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1765 [2026-10-22 01:05:30][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,604 | MFU: 1.8% [2026-10-22 01:05:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9733 [2026-10-22 01:05:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9390 [2026-10-22 01:05:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0702 [2026-10-22 01:05:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1892 [2026-10-22 01:05:36][INFO][training.pretrain_engine] [Ph2|Step 459] loss=4.9811 | ppl=145.63 | aux=0.02505 | lr=2.98e-04 | gnorm=0.477 | tok/s=29,319 | MFU=7.0% | step=143059ms | tokens=1.283B | mem=4.2/32.7GB | cost=$37.63/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 11:02 (130.0h) | Phase3: SKIPPED | SFT: Oct 27 15:47 (134.7h) | DPO (Final): Oct 27 16:21 (135.3h) [2026-10-22 01:05:39][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step459 | Tokens: 1.283B | Spent: $37.63 | Remaining: $962.37 [2026-10-22 01:05:39][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step459 [2026-10-22 01:05:39][INFO][training.pretrain_engine] Starting global step 460 (accumulating 32 micro-batches)... [2026-10-22 01:05:39][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 01:05:39][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-22 01:05:39][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-22 01:05:39][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-22 01:05:39][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step459 to Phase-Technologies/forge-500m-pretrain (path: phase2_step459)... [2026-10-22 01:05:39][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-22 01:05:48][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-22 01:05:48][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step459 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-22 01:05:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2019 [2026-10-22 01:05:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1989 [2026-10-22 01:05:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8840 [2026-10-22 01:05:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1116 [2026-10-22 01:06:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8278 [2026-10-22 01:06:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8606 [2026-10-22 01:06:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4479 [2026-10-22 01:06:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8156 [2026-10-22 01:06:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2800 [2026-10-22 01:06:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4476 [2026-10-22 01:06:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1208 [2026-10-22 01:06:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0782 [2026-10-22 01:06:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2266 [2026-10-22 01:06:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5220 [2026-10-22 01:06:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6166 [2026-10-22 01:06:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2135 [2026-10-22 01:07:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4542 [2026-10-22 01:07:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9103 [2026-10-22 01:07:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8349 [2026-10-22 01:07:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9620 [2026-10-22 01:07:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7911 [2026-10-22 01:07:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2966 [2026-10-22 01:07:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=3.8936 [2026-10-22 01:07:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4818 [2026-10-22 01:07:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7765 [2026-10-22 01:07:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2157 [2026-10-22 01:07:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9593 [2026-10-22 01:07:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8976 [2026-10-22 01:07:57][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,605 | MFU: 1.8% [2026-10-22 01:08:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7755 [2026-10-22 01:08:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6911 [2026-10-22 01:08:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5857 [2026-10-22 01:08:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5754 [2026-10-22 01:08:01][INFO][training.pretrain_engine] [Ph2|Step 460] loss=5.1174 | ppl=166.90 | aux=0.02506 | lr=2.98e-04 | gnorm=0.551 | tok/s=29,320 | MFU=7.0% | step=143051ms | tokens=1.285B | mem=4.2/32.7GB | cost=$37.72/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 11:04 (129.9h) | Phase3: SKIPPED | SFT: Oct 27 15:48 (134.7h) | DPO (Final): Oct 27 16:22 (135.2h) [2026-10-22 01:08:01][INFO][training.pretrain_engine] Starting global step 461 (accumulating 32 micro-batches)... [2026-10-22 01:08:01][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 01:08:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9672 [2026-10-22 01:08:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5101 [2026-10-22 01:08:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8183 [2026-10-22 01:08:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4309 [2026-10-22 01:08:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8545 [2026-10-22 01:08:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6777 [2026-10-22 01:08:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7654 [2026-10-22 01:08:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5697 [2026-10-22 01:08:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1193 [2026-10-22 01:08:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1094 [2026-10-22 01:08:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0674 [2026-10-22 01:08:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1821 [2026-10-22 01:09:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0029 [2026-10-22 01:09:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1201 [2026-10-22 01:09:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0818 [2026-10-22 01:09:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0670 [2026-10-22 01:09:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8220 [2026-10-22 01:09:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3532 [2026-10-22 01:09:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1869 [2026-10-22 01:09:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0906 [2026-10-22 01:09:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0321 [2026-10-22 01:09:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3017 [2026-10-22 01:09:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6867 [2026-10-22 01:09:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9922 [2026-10-22 01:10:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3435 [2026-10-22 01:10:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9488 [2026-10-22 01:10:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6309 [2026-10-22 01:10:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5364 [2026-10-22 01:10:19][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,608 | MFU: 1.8% [2026-10-22 01:10:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1941 [2026-10-22 01:10:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5111 [2026-10-22 01:10:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0837 [2026-10-22 01:10:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9699 [2026-10-22 01:10:24][INFO][training.pretrain_engine] [Ph2|Step 461] loss=4.8561 | ppl=128.52 | aux=0.02506 | lr=2.98e-04 | gnorm=0.512 | tok/s=29,319 | MFU=7.0% | step=143057ms | tokens=1.286B | mem=4.2/32.7GB | cost=$37.80/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 11:06 (129.9h) | Phase3: SKIPPED | SFT: Oct 27 15:50 (134.7h) | DPO (Final): Oct 27 16:24 (135.2h) [2026-10-22 01:10:24][INFO][training.pretrain_engine] Starting global step 462 (accumulating 32 micro-batches)... [2026-10-22 01:10:24][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 01:10:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1605 [2026-10-22 01:10:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1988 [2026-10-22 01:10:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1936 [2026-10-22 01:10:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9017 [2026-10-22 01:11:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7187 [2026-10-22 01:11:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9808 [2026-10-22 01:11:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9405 [2026-10-22 01:11:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0122 [2026-10-22 01:11:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3975 [2026-10-22 01:11:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2089 [2026-10-22 01:11:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6228 [2026-10-22 01:11:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5135 [2026-10-22 01:11:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8722 [2026-10-22 01:11:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2668 [2026-10-22 01:11:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2295 [2026-10-22 01:11:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0757 [2026-10-22 01:11:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2933 [2026-10-22 01:11:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3243 [2026-10-22 01:11:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8035 [2026-10-22 01:11:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2596 [2026-10-22 01:12:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9981 [2026-10-22 01:12:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0851 [2026-10-22 01:12:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1432 [2026-10-22 01:12:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8107 [2026-10-22 01:12:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7846 [2026-10-22 01:12:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3853 [2026-10-22 01:12:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2977 [2026-10-22 01:12:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8737 [2026-10-22 01:12:42][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,601 | MFU: 1.8% [2026-10-22 01:12:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3341 [2026-10-22 01:12:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9794 [2026-10-22 01:12:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2704 [2026-10-22 01:12:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9956 [2026-10-22 01:12:48][INFO][training.pretrain_engine] [Ph2|Step 462] loss=5.0212 | ppl=151.59 | aux=0.02506 | lr=2.98e-04 | gnorm=0.504 | tok/s=29,318 | MFU=7.0% | step=143061ms | tokens=1.287B | mem=4.2/32.7GB | cost=$37.88/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 11:08 (129.9h) | Phase3: SKIPPED | SFT: Oct 27 15:52 (134.7h) | DPO (Final): Oct 27 16:26 (135.2h) [2026-10-22 01:12:48][INFO][training.pretrain_engine] Starting global step 463 (accumulating 32 micro-batches)... [2026-10-22 01:12:48][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 01:13:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7748 [2026-10-22 01:13:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8068 [2026-10-22 01:13:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9639 [2026-10-22 01:13:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0476 [2026-10-22 01:13:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0719 [2026-10-22 01:13:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0648 [2026-10-22 01:13:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0185 [2026-10-22 01:13:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3831 [2026-10-22 01:13:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7455 [2026-10-22 01:13:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9411 [2026-10-22 01:13:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1079 [2026-10-22 01:13:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8550 [2026-10-22 01:13:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0086 [2026-10-22 01:13:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4069 [2026-10-22 01:13:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1367 [2026-10-22 01:13:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6237 [2026-10-22 01:14:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6959 [2026-10-22 01:14:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9533 [2026-10-22 01:14:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8616 [2026-10-22 01:14:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6721 [2026-10-22 01:14:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3156 [2026-10-22 01:14:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8158 [2026-10-22 01:14:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5993 [2026-10-22 01:14:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9961 [2026-10-22 01:14:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5762 [2026-10-22 01:14:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1306 [2026-10-22 01:14:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0543 [2026-10-22 01:14:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1902 [2026-10-22 01:15:06][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,573 | MFU: 1.8% [2026-10-22 01:15:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2853 [2026-10-22 01:15:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8148 [2026-10-22 01:15:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3208 [2026-10-22 01:15:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9987 [2026-10-22 01:15:11][INFO][training.pretrain_engine] [Ph2|Step 463] loss=5.0430 | ppl=154.93 | aux=0.02505 | lr=2.98e-04 | gnorm=0.463 | tok/s=29,320 | MFU=7.0% | step=143054ms | tokens=1.288B | mem=4.2/32.7GB | cost=$37.96/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 11:09 (129.9h) | Phase3: SKIPPED | SFT: Oct 27 15:54 (134.6h) | DPO (Final): Oct 27 16:28 (135.2h) [2026-10-22 01:15:11][INFO][training.pretrain_engine] Starting global step 464 (accumulating 32 micro-batches)... [2026-10-22 01:15:11][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 01:15:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9534 [2026-10-22 01:15:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5923 [2026-10-22 01:15:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9453 [2026-10-22 01:15:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1279 [2026-10-22 01:15:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5162 [2026-10-22 01:15:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1200 [2026-10-22 01:15:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0438 [2026-10-22 01:15:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0987 [2026-10-22 01:16:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7734 [2026-10-22 01:16:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7446 [2026-10-22 01:16:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8854 [2026-10-22 01:16:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6903 [2026-10-22 01:16:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9823 [2026-10-22 01:16:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0987 [2026-10-22 01:16:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9783 [2026-10-22 01:16:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9901 [2026-10-22 01:16:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4459 [2026-10-22 01:16:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8425 [2026-10-22 01:16:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8237 [2026-10-22 01:16:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8863 [2026-10-22 01:16:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0765 [2026-10-22 01:16:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8928 [2026-10-22 01:16:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0551 [2026-10-22 01:16:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0666 [2026-10-22 01:17:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3876 [2026-10-22 01:17:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6951 [2026-10-22 01:17:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2366 [2026-10-22 01:17:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2495 [2026-10-22 01:17:29][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,613 | MFU: 1.8% [2026-10-22 01:17:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0510 [2026-10-22 01:17:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3638 [2026-10-22 01:17:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2955 [2026-10-22 01:17:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9304 [2026-10-22 01:17:34][INFO][training.pretrain_engine] [Ph2|Step 464] loss=5.0083 | ppl=149.64 | aux=0.02506 | lr=2.98e-04 | gnorm=0.516 | tok/s=29,321 | MFU=7.0% | step=143050ms | tokens=1.289B | mem=4.2/32.7GB | cost=$38.04/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 11:11 (129.9h) | Phase3: SKIPPED | SFT: Oct 27 15:55 (134.6h) | DPO (Final): Oct 27 16:29 (135.2h) [2026-10-22 01:17:34][INFO][training.pretrain_engine] Starting global step 465 (accumulating 32 micro-batches)... [2026-10-22 01:17:34][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 01:17:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9447 [2026-10-22 01:17:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5881 [2026-10-22 01:17:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9082 [2026-10-22 01:17:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1568 [2026-10-22 01:18:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6163 [2026-10-22 01:18:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0470 [2026-10-22 01:18:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1536 [2026-10-22 01:18:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2603 [2026-10-22 01:18:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4917 [2026-10-22 01:18:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5742 [2026-10-22 01:18:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7916 [2026-10-22 01:18:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8205 [2026-10-22 01:18:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7115 [2026-10-22 01:18:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6901 [2026-10-22 01:18:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7425 [2026-10-22 01:18:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8342 [2026-10-22 01:19:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7033 [2026-10-22 01:19:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9564 [2026-10-22 01:19:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0164 [2026-10-22 01:19:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9392 [2026-10-22 01:19:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8995 [2026-10-22 01:19:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2014 [2026-10-22 01:19:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3738 [2026-10-22 01:19:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7818 [2026-10-22 01:19:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8555 [2026-10-22 01:19:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5286 [2026-10-22 01:19:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5893 [2026-10-22 01:19:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8449 [2026-10-22 01:19:52][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,600 | MFU: 1.8% [2026-10-22 01:19:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9074 [2026-10-22 01:19:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7817 [2026-10-22 01:19:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6632 [2026-10-22 01:19:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3936 [2026-10-22 01:19:57][INFO][training.pretrain_engine] [Ph2|Step 465] loss=5.0243 | ppl=152.06 | aux=0.02506 | lr=2.98e-04 | gnorm=0.477 | tok/s=29,320 | MFU=7.0% | step=143052ms | tokens=1.290B | mem=4.2/32.7GB | cost=$38.12/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 11:13 (129.9h) | Phase3: SKIPPED | SFT: Oct 27 15:57 (134.6h) | DPO (Final): Oct 27 16:31 (135.2h) [2026-10-22 01:19:57][INFO][training.pretrain_engine] Starting global step 466 (accumulating 32 micro-batches)... [2026-10-22 01:19:57][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 01:20:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0968 [2026-10-22 01:20:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4879 [2026-10-22 01:20:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2782 [2026-10-22 01:20:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6859 [2026-10-22 01:20:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0893 [2026-10-22 01:20:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9788 [2026-10-22 01:20:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4494 [2026-10-22 01:20:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1549 [2026-10-22 01:20:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2786 [2026-10-22 01:20:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9818 [2026-10-22 01:20:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9326 [2026-10-22 01:20:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9564 [2026-10-22 01:21:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7792 [2026-10-22 01:21:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5164 [2026-10-22 01:21:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9069 [2026-10-22 01:21:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2737 [2026-10-22 01:21:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9181 [2026-10-22 01:21:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2986 [2026-10-22 01:21:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8844 [2026-10-22 01:21:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7770 [2026-10-22 01:21:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1492 [2026-10-22 01:21:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4892 [2026-10-22 01:21:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2088 [2026-10-22 01:21:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9937 [2026-10-22 01:22:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2104 [2026-10-22 01:22:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1698 [2026-10-22 01:22:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8155 [2026-10-22 01:22:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3764 [2026-10-22 01:22:15][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,592 | MFU: 1.8% [2026-10-22 01:22:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0413 [2026-10-22 01:22:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7924 [2026-10-22 01:22:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4241 [2026-10-22 01:22:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8921 [2026-10-22 01:22:20][INFO][training.pretrain_engine] [Ph2|Step 466] loss=4.8952 | ppl=133.64 | aux=0.02506 | lr=2.98e-04 | gnorm=0.570 | tok/s=29,320 | MFU=7.0% | step=143051ms | tokens=1.291B | mem=4.2/32.7GB | cost=$38.21/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 11:15 (129.9h) | Phase3: SKIPPED | SFT: Oct 27 15:59 (134.6h) | DPO (Final): Oct 27 16:33 (135.2h) [2026-10-22 01:22:20][INFO][training.pretrain_engine] Starting global step 467 (accumulating 32 micro-batches)... [2026-10-22 01:22:20][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 01:22:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8823 [2026-10-22 01:22:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3159 [2026-10-22 01:22:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1671 [2026-10-22 01:22:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3009 [2026-10-22 01:22:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0470 [2026-10-22 01:22:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9966 [2026-10-22 01:22:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1605 [2026-10-22 01:22:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9240 [2026-10-22 01:23:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1050 [2026-10-22 01:23:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9226 [2026-10-22 01:23:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9489 [2026-10-22 01:23:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8613 [2026-10-22 01:23:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2333 [2026-10-22 01:23:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6759 [2026-10-22 01:23:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8661 [2026-10-22 01:23:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3870 [2026-10-22 01:23:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3009 [2026-10-22 01:23:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3531 [2026-10-22 01:23:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6327 [2026-10-22 01:23:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8283 [2026-10-22 01:24:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9767 [2026-10-22 01:24:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4045 [2026-10-22 01:24:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9680 [2026-10-22 01:24:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1547 [2026-10-22 01:24:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7159 [2026-10-22 01:24:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0068 [2026-10-22 01:24:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9287 [2026-10-22 01:24:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6118 [2026-10-22 01:24:38][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,579 | MFU: 1.8% [2026-10-22 01:24:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9234 [2026-10-22 01:24:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6415 [2026-10-22 01:24:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7449 [2026-10-22 01:24:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2838 [2026-10-22 01:24:43][INFO][training.pretrain_engine] [Ph2|Step 467] loss=4.9071 | ppl=135.24 | aux=0.02506 | lr=2.98e-04 | gnorm=0.602 | tok/s=29,319 | MFU=7.0% | step=143057ms | tokens=1.292B | mem=4.2/32.7GB | cost=$38.29/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 11:17 (129.9h) | Phase3: SKIPPED | SFT: Oct 27 16:01 (134.6h) | DPO (Final): Oct 27 16:35 (135.2h) [2026-10-22 01:24:43][INFO][training.pretrain_engine] Starting global step 468 (accumulating 32 micro-batches)... [2026-10-22 01:24:43][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 01:25:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3193 [2026-10-22 01:25:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8412 [2026-10-22 01:25:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8459 [2026-10-22 01:25:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9268 [2026-10-22 01:25:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7827 [2026-10-22 01:25:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3260 [2026-10-22 01:25:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8903 [2026-10-22 01:25:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.7571 [2026-10-22 01:25:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=3.7222 [2026-10-22 01:25:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3996 [2026-10-22 01:25:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7209 [2026-10-22 01:25:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0240 [2026-10-22 01:25:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5754 [2026-10-22 01:25:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8788 [2026-10-22 01:25:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8773 [2026-10-22 01:25:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3976 [2026-10-22 01:26:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2319 [2026-10-22 01:26:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3605 [2026-10-22 01:26:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8498 [2026-10-22 01:26:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1194 [2026-10-22 01:26:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7796 [2026-10-22 01:26:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6985 [2026-10-22 01:26:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3122 [2026-10-22 01:26:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0932 [2026-10-22 01:26:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0144 [2026-10-22 01:26:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0335 [2026-10-22 01:26:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1273 [2026-10-22 01:26:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1133 [2026-10-22 01:27:01][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,573 | MFU: 1.8% [2026-10-22 01:27:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0419 [2026-10-22 01:27:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5213 [2026-10-22 01:27:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2257 [2026-10-22 01:27:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5058 [2026-10-22 01:27:06][INFO][training.pretrain_engine] [Ph2|Step 468] loss=5.0272 | ppl=152.51 | aux=0.02506 | lr=2.98e-04 | gnorm=0.691 | tok/s=29,317 | MFU=7.0% | step=143068ms | tokens=1.293B | mem=4.2/32.7GB | cost=$38.37/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 11:19 (129.9h) | Phase3: SKIPPED | SFT: Oct 27 16:03 (134.6h) | DPO (Final): Oct 27 16:37 (135.2h) [2026-10-22 01:27:10][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step468 | Tokens: 1.293B | Spent: $38.37 | Remaining: $961.63 [2026-10-22 01:27:10][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step468 [2026-10-22 01:27:10][INFO][training.pretrain_engine] Starting global step 469 (accumulating 32 micro-batches)... [2026-10-22 01:27:10][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 01:27:10][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-22 01:27:10][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-22 01:27:10][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-22 01:27:10][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step468 to Phase-Technologies/forge-500m-pretrain (path: phase2_step468)... [2026-10-22 01:27:10][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-22 01:27:20][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-22 01:27:20][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step468 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-22 01:27:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0212 [2026-10-22 01:27:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2175 [2026-10-22 01:27:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5911 [2026-10-22 01:27:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4191 [2026-10-22 01:27:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1140 [2026-10-22 01:27:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9281 [2026-10-22 01:27:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3098 [2026-10-22 01:27:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9915 [2026-10-22 01:28:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8527 [2026-10-22 01:28:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3557 [2026-10-22 01:28:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9338 [2026-10-22 01:28:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6955 [2026-10-22 01:28:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=3.9955 [2026-10-22 01:28:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=3.6911 [2026-10-22 01:28:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=3.2078 [2026-10-22 01:28:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=3.9157 [2026-10-22 01:28:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9873 [2026-10-22 01:28:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0297 [2026-10-22 01:28:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1157 [2026-10-22 01:28:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9450 [2026-10-22 01:28:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0966 [2026-10-22 01:28:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1860 [2026-10-22 01:28:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8787 [2026-10-22 01:28:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7543 [2026-10-22 01:29:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7780 [2026-10-22 01:29:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0980 [2026-10-22 01:29:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6484 [2026-10-22 01:29:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.1297 [2026-10-22 01:29:28][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,578 | MFU: 1.8% [2026-10-22 01:29:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1691 [2026-10-22 01:29:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1435 [2026-10-22 01:29:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8820 [2026-10-22 01:29:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8446 [2026-10-22 01:29:33][INFO][training.pretrain_engine] [Ph2|Step 469] loss=4.8427 | ppl=126.81 | aux=0.02507 | lr=2.98e-04 | gnorm=0.742 | tok/s=29,316 | MFU=7.0% | step=143070ms | tokens=1.294B | mem=4.2/32.7GB | cost=$38.45/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 11:21 (129.9h) | Phase3: SKIPPED | SFT: Oct 27 16:05 (134.6h) | DPO (Final): Oct 27 16:39 (135.2h) [2026-10-22 01:29:33][INFO][training.pretrain_engine] Starting global step 470 (accumulating 32 micro-batches)... [2026-10-22 01:29:33][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 01:29:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9249 [2026-10-22 01:29:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5971 [2026-10-22 01:29:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3794 [2026-10-22 01:29:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8356 [2026-10-22 01:30:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0657 [2026-10-22 01:30:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6789 [2026-10-22 01:30:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7463 [2026-10-22 01:30:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1558 [2026-10-22 01:30:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7462 [2026-10-22 01:30:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2376 [2026-10-22 01:30:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2689 [2026-10-22 01:30:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8143 [2026-10-22 01:30:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2063 [2026-10-22 01:30:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7405 [2026-10-22 01:30:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2457 [2026-10-22 01:30:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9125 [2026-10-22 01:31:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1893 [2026-10-22 01:31:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8603 [2026-10-22 01:31:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6029 [2026-10-22 01:31:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2135 [2026-10-22 01:31:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4133 [2026-10-22 01:31:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1934 [2026-10-22 01:31:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1148 [2026-10-22 01:31:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6865 [2026-10-22 01:31:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3915 [2026-10-22 01:31:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2697 [2026-10-22 01:31:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3486 [2026-10-22 01:31:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1071 [2026-10-22 01:31:51][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,582 | MFU: 1.8% [2026-10-22 01:31:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7676 [2026-10-22 01:31:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7029 [2026-10-22 01:31:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5727 [2026-10-22 01:31:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2161 [2026-10-22 01:31:56][INFO][training.pretrain_engine] [Ph2|Step 470] loss=5.0948 | ppl=163.17 | aux=0.02506 | lr=2.98e-04 | gnorm=0.879 | tok/s=29,313 | MFU=7.0% | step=143087ms | tokens=1.295B | mem=4.2/32.7GB | cost=$38.54/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 11:24 (129.9h) | Phase3: SKIPPED | SFT: Oct 27 16:08 (134.6h) | DPO (Final): Oct 27 16:42 (135.2h) [2026-10-22 01:31:56][INFO][training.pretrain_engine] Starting global step 471 (accumulating 32 micro-batches)... [2026-10-22 01:31:56][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 01:32:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7774 [2026-10-22 01:32:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9636 [2026-10-22 01:32:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3476 [2026-10-22 01:32:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8442 [2026-10-22 01:32:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6902 [2026-10-22 01:32:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5813 [2026-10-22 01:32:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8516 [2026-10-22 01:32:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1082 [2026-10-22 01:32:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0724 [2026-10-22 01:32:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8444 [2026-10-22 01:32:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1690 [2026-10-22 01:32:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5722 [2026-10-22 01:33:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2027 [2026-10-22 01:33:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1531 [2026-10-22 01:33:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2012 [2026-10-22 01:33:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8627 [2026-10-22 01:33:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4885 [2026-10-22 01:33:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0215 [2026-10-22 01:33:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1473 [2026-10-22 01:33:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2583 [2026-10-22 01:33:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8146 [2026-10-22 01:33:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0737 [2026-10-22 01:33:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0240 [2026-10-22 01:33:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7924 [2026-10-22 01:34:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6094 [2026-10-22 01:34:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4931 [2026-10-22 01:34:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3809 [2026-10-22 01:34:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8365 [2026-10-22 01:34:15][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,580 | MFU: 1.8% [2026-10-22 01:34:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0971 [2026-10-22 01:34:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5100 [2026-10-22 01:34:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0948 [2026-10-22 01:34:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4049 [2026-10-22 01:34:20][INFO][training.pretrain_engine] [Ph2|Step 471] loss=5.0215 | ppl=151.65 | aux=0.02506 | lr=2.98e-04 | gnorm=1.078 | tok/s=29,310 | MFU=7.0% | step=143100ms | tokens=1.296B | mem=4.2/32.7GB | cost=$38.62/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 11:26 (129.9h) | Phase3: SKIPPED | SFT: Oct 27 16:11 (134.6h) | DPO (Final): Oct 27 16:45 (135.2h) [2026-10-22 01:34:20][INFO][training.pretrain_engine] Starting global step 472 (accumulating 32 micro-batches)... [2026-10-22 01:34:20][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 01:34:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5842 [2026-10-22 01:34:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1595 [2026-10-22 01:34:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8800 [2026-10-22 01:34:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.1288 [2026-10-22 01:34:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5894 [2026-10-22 01:34:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1410 [2026-10-22 01:34:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3891 [2026-10-22 01:34:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5792 [2026-10-22 01:35:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5023 [2026-10-22 01:35:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0203 [2026-10-22 01:35:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6669 [2026-10-22 01:35:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2618 [2026-10-22 01:35:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9921 [2026-10-22 01:35:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6309 [2026-10-22 01:35:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4719 [2026-10-22 01:35:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2916 [2026-10-22 01:35:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0170 [2026-10-22 01:35:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7136 [2026-10-22 01:35:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.1095 [2026-10-22 01:35:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4577 [2026-10-22 01:36:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0781 [2026-10-22 01:36:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1857 [2026-10-22 01:36:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0817 [2026-10-22 01:36:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1871 [2026-10-22 01:36:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1344 [2026-10-22 01:36:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1900 [2026-10-22 01:36:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4289 [2026-10-22 01:36:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9070 [2026-10-22 01:36:38][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,618 | MFU: 1.8% [2026-10-22 01:36:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9908 [2026-10-22 01:36:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4289 [2026-10-22 01:36:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6078 [2026-10-22 01:36:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1556 [2026-10-22 01:36:43][INFO][training.pretrain_engine] [Ph2|Step 472] loss=4.9418 | ppl=140.02 | aux=0.02506 | lr=2.98e-04 | gnorm=0.766 | tok/s=29,309 | MFU=7.0% | step=143104ms | tokens=1.297B | mem=4.2/32.7GB | cost=$38.70/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 11:28 (129.9h) | Phase3: SKIPPED | SFT: Oct 27 16:13 (134.6h) | DPO (Final): Oct 27 16:47 (135.2h) [2026-10-22 01:36:43][INFO][training.pretrain_engine] Starting global step 473 (accumulating 32 micro-batches)... [2026-10-22 01:36:43][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 01:37:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0702 [2026-10-22 01:37:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7858 [2026-10-22 01:37:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4286 [2026-10-22 01:37:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6238 [2026-10-22 01:37:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4954 [2026-10-22 01:37:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2977 [2026-10-22 01:37:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0996 [2026-10-22 01:37:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2317 [2026-10-22 01:37:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3484 [2026-10-22 01:37:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7057 [2026-10-22 01:37:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2713 [2026-10-22 01:37:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0385 [2026-10-22 01:37:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8800 [2026-10-22 01:37:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4051 [2026-10-22 01:37:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5726 [2026-10-22 01:37:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4984 [2026-10-22 01:38:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7039 [2026-10-22 01:38:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=3.4144 [2026-10-22 01:38:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8410 [2026-10-22 01:38:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0247 [2026-10-22 01:38:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6611 [2026-10-22 01:38:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.0374 [2026-10-22 01:38:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5909 [2026-10-22 01:38:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6410 [2026-10-22 01:38:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2077 [2026-10-22 01:38:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=3.5077 [2026-10-22 01:38:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2977 [2026-10-22 01:38:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2199 [2026-10-22 01:39:01][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,615 | MFU: 1.8% [2026-10-22 01:39:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1648 [2026-10-22 01:39:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8634 [2026-10-22 01:39:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1170 [2026-10-22 01:39:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8397 [2026-10-22 01:39:06][INFO][training.pretrain_engine] [Ph2|Step 473] loss=4.9659 | ppl=143.43 | aux=0.02506 | lr=2.98e-04 | gnorm=0.820 | tok/s=29,309 | MFU=7.0% | step=143105ms | tokens=1.298B | mem=4.2/32.7GB | cost=$38.78/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 11:30 (129.9h) | Phase3: SKIPPED | SFT: Oct 27 16:15 (134.6h) | DPO (Final): Oct 27 16:49 (135.2h) [2026-10-22 01:39:06][INFO][training.pretrain_engine] Starting global step 474 (accumulating 32 micro-batches)... [2026-10-22 01:39:06][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 01:39:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2692 [2026-10-22 01:39:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4996 [2026-10-22 01:39:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6426 [2026-10-22 01:39:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5371 [2026-10-22 01:39:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7594 [2026-10-22 01:39:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8539 [2026-10-22 01:39:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0187 [2026-10-22 01:39:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8397 [2026-10-22 01:39:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9838 [2026-10-22 01:40:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0745 [2026-10-22 01:40:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7116 [2026-10-22 01:40:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2491 [2026-10-22 01:40:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7139 [2026-10-22 01:40:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7287 [2026-10-22 01:40:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0900 [2026-10-22 01:40:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0834 [2026-10-22 01:40:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9876 [2026-10-22 01:40:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7505 [2026-10-22 01:40:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9765 [2026-10-22 01:40:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7264 [2026-10-22 01:40:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9429 [2026-10-22 01:40:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=3.7681 [2026-10-22 01:40:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0311 [2026-10-22 01:40:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6907 [2026-10-22 01:41:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1341 [2026-10-22 01:41:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1422 [2026-10-22 01:41:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5762 [2026-10-22 01:41:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1141 [2026-10-22 01:41:24][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,602 | MFU: 1.8% [2026-10-22 01:41:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1030 [2026-10-22 01:41:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0929 [2026-10-22 01:41:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2789 [2026-10-22 01:41:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4058 [2026-10-22 01:41:30][INFO][training.pretrain_engine] [Ph2|Step 474] loss=5.0101 | ppl=149.92 | aux=0.02505 | lr=2.98e-04 | gnorm=0.652 | tok/s=29,308 | MFU=7.0% | step=143109ms | tokens=1.299B | mem=4.2/32.7GB | cost=$38.86/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 11:32 (129.9h) | Phase3: SKIPPED | SFT: Oct 27 16:17 (134.6h) | DPO (Final): Oct 27 16:51 (135.2h) [2026-10-22 01:41:30][INFO][training.pretrain_engine] Starting global step 475 (accumulating 32 micro-batches)... [2026-10-22 01:41:30][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 01:41:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5464 [2026-10-22 01:41:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3544 [2026-10-22 01:41:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7054 [2026-10-22 01:41:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6278 [2026-10-22 01:42:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2852 [2026-10-22 01:42:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4374 [2026-10-22 01:42:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3083 [2026-10-22 01:42:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6423 [2026-10-22 01:42:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3808 [2026-10-22 01:42:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3024 [2026-10-22 01:42:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9133 [2026-10-22 01:42:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9679 [2026-10-22 01:42:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1901 [2026-10-22 01:42:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8934 [2026-10-22 01:42:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3637 [2026-10-22 01:42:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6463 [2026-10-22 01:42:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8373 [2026-10-22 01:42:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0535 [2026-10-22 01:42:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0545 [2026-10-22 01:42:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3651 [2026-10-22 01:43:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0357 [2026-10-22 01:43:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9577 [2026-10-22 01:43:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7733 [2026-10-22 01:43:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2907 [2026-10-22 01:43:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1935 [2026-10-22 01:43:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2287 [2026-10-22 01:43:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7567 [2026-10-22 01:43:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5894 [2026-10-22 01:43:48][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,594 | MFU: 1.8% [2026-10-22 01:43:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9539 [2026-10-22 01:43:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5066 [2026-10-22 01:43:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2153 [2026-10-22 01:43:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7727 [2026-10-22 01:43:53][INFO][training.pretrain_engine] [Ph2|Step 475] loss=5.0678 | ppl=158.82 | aux=0.02506 | lr=2.98e-04 | gnorm=0.793 | tok/s=29,306 | MFU=7.0% | step=143119ms | tokens=1.300B | mem=4.2/32.7GB | cost=$38.95/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 11:35 (129.9h) | Phase3: SKIPPED | SFT: Oct 27 16:19 (134.6h) | DPO (Final): Oct 27 16:53 (135.2h) [2026-10-22 01:43:53][INFO][training.pretrain_engine] Starting global step 476 (accumulating 32 micro-batches)... [2026-10-22 01:43:53][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 01:44:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4512 [2026-10-22 01:44:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6851 [2026-10-22 01:44:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.0607 [2026-10-22 01:44:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.1963 [2026-10-22 01:44:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1569 [2026-10-22 01:44:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9268 [2026-10-22 01:44:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3470 [2026-10-22 01:44:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2794 [2026-10-22 01:44:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2610 [2026-10-22 01:44:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4594 [2026-10-22 01:44:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1715 [2026-10-22 01:44:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1368 [2026-10-22 01:45:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2265 [2026-10-22 01:45:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1639 [2026-10-22 01:45:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5207 [2026-10-22 01:45:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8210 [2026-10-22 01:45:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3381 [2026-10-22 01:45:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5159 [2026-10-22 01:45:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6455 [2026-10-22 01:45:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8162 [2026-10-22 01:45:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4634 [2026-10-22 01:45:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1585 [2026-10-22 01:45:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0991 [2026-10-22 01:45:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1620 [2026-10-22 01:45:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8157 [2026-10-22 01:45:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0661 [2026-10-22 01:45:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1508 [2026-10-22 01:45:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0474 [2026-10-22 01:46:12][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,570 | MFU: 1.8% [2026-10-22 01:46:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2798 [2026-10-22 01:46:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9639 [2026-10-22 01:46:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9267 [2026-10-22 01:46:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8351 [2026-10-22 01:46:17][INFO][training.pretrain_engine] [Ph2|Step 476] loss=4.9948 | ppl=147.64 | aux=0.02505 | lr=2.98e-04 | gnorm=0.656 | tok/s=29,304 | MFU=7.0% | step=143132ms | tokens=1.301B | mem=4.2/32.7GB | cost=$39.03/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 11:37 (129.9h) | Phase3: SKIPPED | SFT: Oct 27 16:21 (134.6h) | DPO (Final): Oct 27 16:56 (135.2h) [2026-10-22 01:46:17][INFO][training.pretrain_engine] Starting global step 477 (accumulating 32 micro-batches)... [2026-10-22 01:46:17][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 01:46:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2018 [2026-10-22 01:46:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8467 [2026-10-22 01:46:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3784 [2026-10-22 01:46:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0384 [2026-10-22 01:46:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9447 [2026-10-22 01:46:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2752 [2026-10-22 01:46:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0542 [2026-10-22 01:46:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8083 [2026-10-22 01:47:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1864 [2026-10-22 01:47:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1705 [2026-10-22 01:47:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0226 [2026-10-22 01:47:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9523 [2026-10-22 01:47:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2458 [2026-10-22 01:47:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0076 [2026-10-22 01:47:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5612 [2026-10-22 01:47:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1671 [2026-10-22 01:47:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2234 [2026-10-22 01:47:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0526 [2026-10-22 01:47:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9138 [2026-10-22 01:47:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8205 [2026-10-22 01:48:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0126 [2026-10-22 01:48:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7793 [2026-10-22 01:48:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8434 [2026-10-22 01:48:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9478 [2026-10-22 01:48:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0914 [2026-10-22 01:48:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6118 [2026-10-22 01:48:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5658 [2026-10-22 01:48:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6979 [2026-10-22 01:48:35][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,605 | MFU: 1.8% [2026-10-22 01:48:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1883 [2026-10-22 01:48:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0173 [2026-10-22 01:48:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0320 [2026-10-22 01:48:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7006 [2026-10-22 01:48:41][INFO][training.pretrain_engine] [Ph2|Step 477] loss=4.9521 | ppl=141.47 | aux=0.02506 | lr=2.98e-04 | gnorm=0.730 | tok/s=29,302 | MFU=7.0% | step=143142ms | tokens=1.302B | mem=4.2/32.7GB | cost=$39.11/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 11:39 (129.9h) | Phase3: SKIPPED | SFT: Oct 27 16:24 (134.6h) | DPO (Final): Oct 27 16:58 (135.2h) [2026-10-22 01:48:44][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step477 | Tokens: 1.302B | Spent: $39.11 | Remaining: $960.89 [2026-10-22 01:48:44][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step477 [2026-10-22 01:48:44][INFO][training.pretrain_engine] Starting global step 478 (accumulating 32 micro-batches)... [2026-10-22 01:48:44][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 01:48:44][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-22 01:48:44][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-22 01:48:44][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-22 01:48:44][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step477 to Phase-Technologies/forge-500m-pretrain (path: phase2_step477)... [2026-10-22 01:48:44][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-22 01:48:54][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-22 01:48:54][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step477 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-22 01:48:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8750 [2026-10-22 01:48:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1228 [2026-10-22 01:48:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9888 [2026-10-22 01:49:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9386 [2026-10-22 01:49:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8955 [2026-10-22 01:49:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6053 [2026-10-22 01:49:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7630 [2026-10-22 01:49:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3801 [2026-10-22 01:49:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2663 [2026-10-22 01:49:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=3.9699 [2026-10-22 01:49:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5665 [2026-10-22 01:49:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7267 [2026-10-22 01:49:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4468 [2026-10-22 01:49:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9211 [2026-10-22 01:49:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8429 [2026-10-22 01:49:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8281 [2026-10-22 01:50:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4722 [2026-10-22 01:50:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0840 [2026-10-22 01:50:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5684 [2026-10-22 01:50:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4192 [2026-10-22 01:50:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=3.0137 [2026-10-22 01:50:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=3.8553 [2026-10-22 01:50:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6740 [2026-10-22 01:50:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3383 [2026-10-22 01:50:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3085 [2026-10-22 01:50:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3134 [2026-10-22 01:50:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9248 [2026-10-22 01:50:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4354 [2026-10-22 01:51:02][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,581 | MFU: 1.8% [2026-10-22 01:51:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2088 [2026-10-22 01:51:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8808 [2026-10-22 01:51:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9275 [2026-10-22 01:51:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5904 [2026-10-22 01:51:06][INFO][training.pretrain_engine] [Ph2|Step 478] loss=4.9772 | ppl=145.07 | aux=0.02506 | lr=2.98e-04 | gnorm=0.523 | tok/s=29,301 | MFU=7.0% | step=143144ms | tokens=1.303B | mem=4.2/32.7GB | cost=$39.19/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 11:41 (129.8h) | Phase3: SKIPPED | SFT: Oct 27 16:26 (134.6h) | DPO (Final): Oct 27 17:00 (135.2h) [2026-10-22 01:51:06][INFO][training.pretrain_engine] Starting global step 479 (accumulating 32 micro-batches)... [2026-10-22 01:51:06][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 01:51:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2071 [2026-10-22 01:51:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0975 [2026-10-22 01:51:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1909 [2026-10-22 01:51:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0435 [2026-10-22 01:51:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1067 [2026-10-22 01:51:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5211 [2026-10-22 01:51:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8146 [2026-10-22 01:51:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9767 [2026-10-22 01:52:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0449 [2026-10-22 01:52:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4222 [2026-10-22 01:52:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1894 [2026-10-22 01:52:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2473 [2026-10-22 01:52:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5132 [2026-10-22 01:52:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7942 [2026-10-22 01:52:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0866 [2026-10-22 01:52:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5781 [2026-10-22 01:52:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9224 [2026-10-22 01:52:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3206 [2026-10-22 01:52:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9037 [2026-10-22 01:52:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8955 [2026-10-22 01:52:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0080 [2026-10-22 01:52:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7152 [2026-10-22 01:52:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1039 [2026-10-22 01:52:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6236 [2026-10-22 01:53:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8532 [2026-10-22 01:53:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2019 [2026-10-22 01:53:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6373 [2026-10-22 01:53:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9700 [2026-10-22 01:53:25][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,574 | MFU: 1.8% [2026-10-22 01:53:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0733 [2026-10-22 01:53:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1125 [2026-10-22 01:53:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7325 [2026-10-22 01:53:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8137 [2026-10-22 01:53:30][INFO][training.pretrain_engine] [Ph2|Step 479] loss=4.9706 | ppl=144.12 | aux=0.02506 | lr=2.98e-04 | gnorm=0.613 | tok/s=29,298 | MFU=7.0% | step=143158ms | tokens=1.304B | mem=4.2/32.7GB | cost=$39.28/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 11:44 (129.8h) | Phase3: SKIPPED | SFT: Oct 27 16:28 (134.6h) | DPO (Final): Oct 27 17:02 (135.2h) [2026-10-22 01:53:30][INFO][training.pretrain_engine] Starting global step 480 (accumulating 32 micro-batches)... [2026-10-22 01:53:30][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 01:53:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3820 [2026-10-22 01:53:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4905 [2026-10-22 01:53:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1481 [2026-10-22 01:53:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2179 [2026-10-22 01:54:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5480 [2026-10-22 01:54:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9012 [2026-10-22 01:54:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4995 [2026-10-22 01:54:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8120 [2026-10-22 01:54:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8170 [2026-10-22 01:54:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3860 [2026-10-22 01:54:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9012 [2026-10-22 01:54:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8229 [2026-10-22 01:54:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.1895 [2026-10-22 01:54:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7268 [2026-10-22 01:54:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4399 [2026-10-22 01:54:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6927 [2026-10-22 01:54:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7551 [2026-10-22 01:54:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7151 [2026-10-22 01:55:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4554 [2026-10-22 01:55:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2142 [2026-10-22 01:55:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5046 [2026-10-22 01:55:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5536 [2026-10-22 01:55:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0654 [2026-10-22 01:55:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8017 [2026-10-22 01:55:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6779 [2026-10-22 01:55:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3095 [2026-10-22 01:55:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7037 [2026-10-22 01:55:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2576 [2026-10-22 01:55:49][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,584 | MFU: 1.8% [2026-10-22 01:55:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2663 [2026-10-22 01:55:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8183 [2026-10-22 01:55:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.1716 [2026-10-22 01:55:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9124 [2026-10-22 01:55:55][INFO][training.pretrain_engine] [Ph2|Step 480] loss=5.0080 | ppl=149.60 | aux=0.02508 | lr=2.98e-04 | gnorm=0.559 | tok/s=29,295 | MFU=7.0% | step=143172ms | tokens=1.305B | mem=4.2/32.7GB | cost=$39.36/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 11:46 (129.9h) | Phase3: SKIPPED | SFT: Oct 27 16:31 (134.6h) | DPO (Final): Oct 27 17:05 (135.2h) [2026-10-22 01:55:55][INFO][training.pretrain_engine] Starting global step 481 (accumulating 32 micro-batches)... [2026-10-22 01:55:55][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 01:56:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9264 [2026-10-22 01:56:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9062 [2026-10-22 01:56:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9635 [2026-10-22 01:56:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.2931 [2026-10-22 01:56:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9412 [2026-10-22 01:56:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8899 [2026-10-22 01:56:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8162 [2026-10-22 01:56:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3335 [2026-10-22 01:56:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8095 [2026-10-22 01:56:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0413 [2026-10-22 01:56:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3072 [2026-10-22 01:56:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9948 [2026-10-22 01:57:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1102 [2026-10-22 01:57:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1042 [2026-10-22 01:57:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3295 [2026-10-22 01:57:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2069 [2026-10-22 01:57:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2442 [2026-10-22 01:57:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0185 [2026-10-22 01:57:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2244 [2026-10-22 01:57:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5147 [2026-10-22 01:57:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9320 [2026-10-22 01:57:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1019 [2026-10-22 01:57:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8673 [2026-10-22 01:57:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2690 [2026-10-22 01:57:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0208 [2026-10-22 01:57:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0184 [2026-10-22 01:57:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9185 [2026-10-22 01:58:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6778 [2026-10-22 01:58:12][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,615 | MFU: 1.8% [2026-10-22 01:58:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9607 [2026-10-22 01:58:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6121 [2026-10-22 01:58:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4351 [2026-10-22 01:58:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6510 [2026-10-22 01:58:18][INFO][training.pretrain_engine] [Ph2|Step 481] loss=4.9062 | ppl=135.13 | aux=0.02507 | lr=2.98e-04 | gnorm=0.559 | tok/s=29,294 | MFU=7.0% | step=143178ms | tokens=1.307B | mem=4.2/32.7GB | cost=$39.44/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 11:49 (129.8h) | Phase3: SKIPPED | SFT: Oct 27 16:33 (134.6h) | DPO (Final): Oct 27 17:07 (135.2h) [2026-10-22 01:58:18][INFO][training.pretrain_engine] Starting global step 482 (accumulating 32 micro-batches)... [2026-10-22 01:58:18][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 01:58:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0612 [2026-10-22 01:58:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3360 [2026-10-22 01:58:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2219 [2026-10-22 01:58:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2663 [2026-10-22 01:58:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1049 [2026-10-22 01:58:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8292 [2026-10-22 01:58:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6983 [2026-10-22 01:58:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2224 [2026-10-22 01:59:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9541 [2026-10-22 01:59:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4742 [2026-10-22 01:59:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3954 [2026-10-22 01:59:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1246 [2026-10-22 01:59:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9572 [2026-10-22 01:59:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8309 [2026-10-22 01:59:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0687 [2026-10-22 01:59:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0917 [2026-10-22 01:59:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8483 [2026-10-22 01:59:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8611 [2026-10-22 01:59:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1480 [2026-10-22 01:59:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7975 [2026-10-22 02:00:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2055 [2026-10-22 02:00:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8078 [2026-10-22 02:00:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3797 [2026-10-22 02:00:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3478 [2026-10-22 02:00:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8686 [2026-10-22 02:00:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6150 [2026-10-22 02:00:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1631 [2026-10-22 02:00:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7256 [2026-10-22 02:00:36][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,622 | MFU: 1.8% [2026-10-22 02:00:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7318 [2026-10-22 02:00:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9281 [2026-10-22 02:00:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1283 [2026-10-22 02:00:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8415 [2026-10-22 02:00:41][INFO][training.pretrain_engine] [Ph2|Step 482] loss=4.9499 | ppl=141.16 | aux=0.02506 | lr=2.98e-04 | gnorm=0.562 | tok/s=29,297 | MFU=7.0% | step=143167ms | tokens=1.308B | mem=4.2/32.7GB | cost=$39.52/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 11:50 (129.8h) | Phase3: SKIPPED | SFT: Oct 27 16:34 (134.6h) | DPO (Final): Oct 27 17:08 (135.1h) [2026-10-22 02:00:41][INFO][training.pretrain_engine] Starting global step 483 (accumulating 32 micro-batches)... [2026-10-22 02:00:41][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 02:00:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9688 [2026-10-22 02:00:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9909 [2026-10-22 02:00:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2919 [2026-10-22 02:00:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0182 [2026-10-22 02:01:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0734 [2026-10-22 02:01:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9069 [2026-10-22 02:01:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8947 [2026-10-22 02:01:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0125 [2026-10-22 02:01:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6017 [2026-10-22 02:01:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3842 [2026-10-22 02:01:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3028 [2026-10-22 02:01:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1125 [2026-10-22 02:01:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7653 [2026-10-22 02:01:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6638 [2026-10-22 02:01:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7110 [2026-10-22 02:01:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8299 [2026-10-22 02:02:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2450 [2026-10-22 02:02:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2305 [2026-10-22 02:02:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7876 [2026-10-22 02:02:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0505 [2026-10-22 02:02:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5760 [2026-10-22 02:02:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5370 [2026-10-22 02:02:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0738 [2026-10-22 02:02:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5691 [2026-10-22 02:02:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1247 [2026-10-22 02:02:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8937 [2026-10-22 02:02:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9574 [2026-10-22 02:02:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0398 [2026-10-22 02:02:59][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,599 | MFU: 1.8% [2026-10-22 02:03:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8212 [2026-10-22 02:03:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1338 [2026-10-22 02:03:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4880 [2026-10-22 02:03:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1078 [2026-10-22 02:03:04][INFO][training.pretrain_engine] [Ph2|Step 483] loss=5.0571 | ppl=157.13 | aux=0.02506 | lr=2.98e-04 | gnorm=0.486 | tok/s=29,297 | MFU=7.0% | step=143164ms | tokens=1.309B | mem=4.2/32.7GB | cost=$39.61/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 11:51 (129.8h) | Phase3: SKIPPED | SFT: Oct 27 16:36 (134.6h) | DPO (Final): Oct 27 17:10 (135.1h) [2026-10-22 02:03:04][INFO][training.pretrain_engine] Starting global step 484 (accumulating 32 micro-batches)... [2026-10-22 02:03:04][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 02:03:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8866 [2026-10-22 02:03:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.9790 [2026-10-22 02:03:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3118 [2026-10-22 02:03:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5383 [2026-10-22 02:03:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6509 [2026-10-22 02:03:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3140 [2026-10-22 02:03:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8246 [2026-10-22 02:03:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2434 [2026-10-22 02:03:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0346 [2026-10-22 02:03:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2780 [2026-10-22 02:03:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.0793 [2026-10-22 02:03:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1327 [2026-10-22 02:04:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1169 [2026-10-22 02:04:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=3.6041 [2026-10-22 02:04:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6155 [2026-10-22 02:04:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8018 [2026-10-22 02:04:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6104 [2026-10-22 02:04:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2642 [2026-10-22 02:04:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5253 [2026-10-22 02:04:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1273 [2026-10-22 02:04:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1288 [2026-10-22 02:04:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2859 [2026-10-22 02:04:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1958 [2026-10-22 02:04:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2098 [2026-10-22 02:05:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4886 [2026-10-22 02:05:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1500 [2026-10-22 02:05:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4288 [2026-10-22 02:05:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2087 [2026-10-22 02:05:22][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,596 | MFU: 1.8% [2026-10-22 02:05:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3861 [2026-10-22 02:05:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4511 [2026-10-22 02:05:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1721 [2026-10-22 02:05:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9984 [2026-10-22 02:05:27][INFO][training.pretrain_engine] [Ph2|Step 484] loss=4.9314 | ppl=138.57 | aux=0.02506 | lr=2.98e-04 | gnorm=0.547 | tok/s=29,296 | MFU=7.0% | step=143169ms | tokens=1.310B | mem=4.2/32.7GB | cost=$39.69/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 11:53 (129.8h) | Phase3: SKIPPED | SFT: Oct 27 16:38 (134.5h) | DPO (Final): Oct 27 17:12 (135.1h) [2026-10-22 02:05:27][INFO][training.pretrain_engine] Starting global step 485 (accumulating 32 micro-batches)... [2026-10-22 02:05:27][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 02:05:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2605 [2026-10-22 02:05:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1425 [2026-10-22 02:05:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9770 [2026-10-22 02:05:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0930 [2026-10-22 02:06:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9495 [2026-10-22 02:06:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5661 [2026-10-22 02:06:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5671 [2026-10-22 02:06:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6693 [2026-10-22 02:06:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1076 [2026-10-22 02:06:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7194 [2026-10-22 02:06:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1761 [2026-10-22 02:06:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6058 [2026-10-22 02:06:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5554 [2026-10-22 02:06:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3309 [2026-10-22 02:06:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1612 [2026-10-22 02:06:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1665 [2026-10-22 02:06:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8148 [2026-10-22 02:06:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5284 [2026-10-22 02:06:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4548 [2026-10-22 02:06:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3681 [2026-10-22 02:07:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6648 [2026-10-22 02:07:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8972 [2026-10-22 02:07:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9671 [2026-10-22 02:07:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6195 [2026-10-22 02:07:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5130 [2026-10-22 02:07:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3328 [2026-10-22 02:07:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1900 [2026-10-22 02:07:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1720 [2026-10-22 02:07:46][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,573 | MFU: 1.8% [2026-10-22 02:07:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8714 [2026-10-22 02:07:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6093 [2026-10-22 02:07:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7503 [2026-10-22 02:07:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1289 [2026-10-22 02:07:51][INFO][training.pretrain_engine] [Ph2|Step 485] loss=5.0156 | ppl=150.75 | aux=0.02505 | lr=2.98e-04 | gnorm=0.473 | tok/s=29,296 | MFU=7.0% | step=143172ms | tokens=1.311B | mem=4.2/32.7GB | cost=$39.77/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 11:55 (129.8h) | Phase3: SKIPPED | SFT: Oct 27 16:40 (134.5h) | DPO (Final): Oct 27 17:14 (135.1h) [2026-10-22 02:07:51][INFO][training.pretrain_engine] Starting global step 486 (accumulating 32 micro-batches)... [2026-10-22 02:07:51][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 02:08:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8047 [2026-10-22 02:08:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8101 [2026-10-22 02:08:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7204 [2026-10-22 02:08:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3161 [2026-10-22 02:08:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2035 [2026-10-22 02:08:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5532 [2026-10-22 02:08:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3441 [2026-10-22 02:08:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6085 [2026-10-22 02:08:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9230 [2026-10-22 02:08:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4842 [2026-10-22 02:08:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9610 [2026-10-22 02:08:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6858 [2026-10-22 02:09:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8082 [2026-10-22 02:09:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5044 [2026-10-22 02:09:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4924 [2026-10-22 02:09:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7312 [2026-10-22 02:09:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7703 [2026-10-22 02:09:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6020 [2026-10-22 02:09:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4457 [2026-10-22 02:09:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2793 [2026-10-22 02:09:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.1952 [2026-10-22 02:09:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0287 [2026-10-22 02:09:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3516 [2026-10-22 02:09:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9473 [2026-10-22 02:09:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3257 [2026-10-22 02:09:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2728 [2026-10-22 02:09:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5359 [2026-10-22 02:09:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2466 [2026-10-22 02:10:09][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,577 | MFU: 1.8% [2026-10-22 02:10:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3796 [2026-10-22 02:10:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6681 [2026-10-22 02:10:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6098 [2026-10-22 02:10:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6092 [2026-10-22 02:10:14][INFO][training.pretrain_engine] [Ph2|Step 486] loss=4.9499 | ppl=141.16 | aux=0.02507 | lr=2.98e-04 | gnorm=0.498 | tok/s=29,294 | MFU=7.0% | step=143179ms | tokens=1.312B | mem=4.2/32.7GB | cost=$39.85/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 11:58 (129.8h) | Phase3: SKIPPED | SFT: Oct 27 16:42 (134.5h) | DPO (Final): Oct 27 17:16 (135.1h) [2026-10-22 02:10:18][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step486 | Tokens: 1.312B | Spent: $39.85 | Remaining: $960.15 [2026-10-22 02:10:18][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step486 [2026-10-22 02:10:18][INFO][training.pretrain_engine] Starting global step 487 (accumulating 32 micro-batches)... [2026-10-22 02:10:18][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 02:10:18][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-22 02:10:18][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-22 02:10:18][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-22 02:10:18][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step486 to Phase-Technologies/forge-500m-pretrain (path: phase2_step486)... [2026-10-22 02:10:18][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-22 02:10:27][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-22 02:10:27][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step486 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-22 02:10:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8651 [2026-10-22 02:10:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3515 [2026-10-22 02:10:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2579 [2026-10-22 02:10:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3831 [2026-10-22 02:10:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9109 [2026-10-22 02:10:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7885 [2026-10-22 02:10:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0775 [2026-10-22 02:10:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9679 [2026-10-22 02:11:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9397 [2026-10-22 02:11:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1590 [2026-10-22 02:11:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0980 [2026-10-22 02:11:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9969 [2026-10-22 02:11:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8384 [2026-10-22 02:11:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1851 [2026-10-22 02:11:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9433 [2026-10-22 02:11:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6657 [2026-10-22 02:11:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0744 [2026-10-22 02:11:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8439 [2026-10-22 02:11:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7209 [2026-10-22 02:11:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2751 [2026-10-22 02:12:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5821 [2026-10-22 02:12:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8213 [2026-10-22 02:12:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0156 [2026-10-22 02:12:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8728 [2026-10-22 02:12:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.1859 [2026-10-22 02:12:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7079 [2026-10-22 02:12:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7016 [2026-10-22 02:12:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9553 [2026-10-22 02:12:36][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,591 | MFU: 1.8% [2026-10-22 02:12:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7613 [2026-10-22 02:12:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8542 [2026-10-22 02:12:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9415 [2026-10-22 02:12:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.0439 [2026-10-22 02:12:40][INFO][training.pretrain_engine] [Ph2|Step 487] loss=4.9007 | ppl=134.39 | aux=0.02507 | lr=2.98e-04 | gnorm=0.459 | tok/s=29,295 | MFU=7.0% | step=143177ms | tokens=1.313B | mem=4.2/32.7GB | cost=$39.94/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 11:59 (129.8h) | Phase3: SKIPPED | SFT: Oct 27 16:44 (134.5h) | DPO (Final): Oct 27 17:18 (135.1h) [2026-10-22 02:12:40][INFO][training.pretrain_engine] Starting global step 488 (accumulating 32 micro-batches)... [2026-10-22 02:12:40][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 02:12:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0036 [2026-10-22 02:12:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3533 [2026-10-22 02:12:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1406 [2026-10-22 02:12:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2493 [2026-10-22 02:13:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6160 [2026-10-22 02:13:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8228 [2026-10-22 02:13:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9771 [2026-10-22 02:13:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7712 [2026-10-22 02:13:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8522 [2026-10-22 02:13:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0663 [2026-10-22 02:13:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1928 [2026-10-22 02:13:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1868 [2026-10-22 02:13:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9127 [2026-10-22 02:13:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8229 [2026-10-22 02:13:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5670 [2026-10-22 02:13:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9336 [2026-10-22 02:14:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9369 [2026-10-22 02:14:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3167 [2026-10-22 02:14:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0675 [2026-10-22 02:14:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0895 [2026-10-22 02:14:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5811 [2026-10-22 02:14:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5908 [2026-10-22 02:14:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4085 [2026-10-22 02:14:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4254 [2026-10-22 02:14:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7819 [2026-10-22 02:14:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2800 [2026-10-22 02:14:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4607 [2026-10-22 02:14:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9872 [2026-10-22 02:14:59][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,581 | MFU: 1.8% [2026-10-22 02:15:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9739 [2026-10-22 02:15:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6172 [2026-10-22 02:15:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2595 [2026-10-22 02:15:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2578 [2026-10-22 02:15:03][INFO][training.pretrain_engine] [Ph2|Step 488] loss=4.9295 | ppl=138.31 | aux=0.02506 | lr=2.98e-04 | gnorm=0.463 | tok/s=29,293 | MFU=7.0% | step=143185ms | tokens=1.314B | mem=4.2/32.7GB | cost=$40.02/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 12:01 (129.8h) | Phase3: SKIPPED | SFT: Oct 27 16:46 (134.5h) | DPO (Final): Oct 27 17:20 (135.1h) [2026-10-22 02:15:03][INFO][training.pretrain_engine] Starting global step 489 (accumulating 32 micro-batches)... [2026-10-22 02:15:03][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 02:15:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8154 [2026-10-22 02:15:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6095 [2026-10-22 02:15:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.1390 [2026-10-22 02:15:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4757 [2026-10-22 02:15:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1025 [2026-10-22 02:15:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0222 [2026-10-22 02:15:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2290 [2026-10-22 02:15:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1542 [2026-10-22 02:15:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9361 [2026-10-22 02:15:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7612 [2026-10-22 02:15:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7089 [2026-10-22 02:15:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2206 [2026-10-22 02:16:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5378 [2026-10-22 02:16:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.2516 [2026-10-22 02:16:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1231 [2026-10-22 02:16:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0756 [2026-10-22 02:16:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7428 [2026-10-22 02:16:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1033 [2026-10-22 02:16:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6794 [2026-10-22 02:16:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3681 [2026-10-22 02:16:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9171 [2026-10-22 02:16:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3415 [2026-10-22 02:16:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1976 [2026-10-22 02:16:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3680 [2026-10-22 02:17:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7639 [2026-10-22 02:17:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1563 [2026-10-22 02:17:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7507 [2026-10-22 02:17:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1209 [2026-10-22 02:17:21][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,599 | MFU: 1.8% [2026-10-22 02:17:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4188 [2026-10-22 02:17:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3493 [2026-10-22 02:17:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7187 [2026-10-22 02:17:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1629 [2026-10-22 02:17:26][INFO][training.pretrain_engine] [Ph2|Step 489] loss=4.9042 | ppl=134.86 | aux=0.02506 | lr=2.98e-04 | gnorm=0.566 | tok/s=29,293 | MFU=7.0% | step=143184ms | tokens=1.315B | mem=4.2/32.7GB | cost=$40.10/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 12:03 (129.8h) | Phase3: SKIPPED | SFT: Oct 27 16:48 (134.5h) | DPO (Final): Oct 27 17:22 (135.1h) [2026-10-22 02:17:26][INFO][training.pretrain_engine] Starting global step 490 (accumulating 32 micro-batches)... [2026-10-22 02:17:26][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 02:17:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5696 [2026-10-22 02:17:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1037 [2026-10-22 02:17:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7959 [2026-10-22 02:17:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2098 [2026-10-22 02:18:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5481 [2026-10-22 02:18:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6573 [2026-10-22 02:18:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7843 [2026-10-22 02:18:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3310 [2026-10-22 02:18:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2775 [2026-10-22 02:18:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9947 [2026-10-22 02:18:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4088 [2026-10-22 02:18:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9069 [2026-10-22 02:18:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3915 [2026-10-22 02:18:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8067 [2026-10-22 02:18:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4936 [2026-10-22 02:18:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3858 [2026-10-22 02:18:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0385 [2026-10-22 02:18:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7617 [2026-10-22 02:18:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4708 [2026-10-22 02:18:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8772 [2026-10-22 02:19:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3610 [2026-10-22 02:19:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6015 [2026-10-22 02:19:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8421 [2026-10-22 02:19:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7910 [2026-10-22 02:19:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4790 [2026-10-22 02:19:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2554 [2026-10-22 02:19:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5824 [2026-10-22 02:19:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0564 [2026-10-22 02:19:44][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,609 | MFU: 1.8% [2026-10-22 02:19:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8761 [2026-10-22 02:19:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8268 [2026-10-22 02:19:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0683 [2026-10-22 02:19:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1109 [2026-10-22 02:19:50][INFO][training.pretrain_engine] [Ph2|Step 490] loss=4.9737 | ppl=144.56 | aux=0.02506 | lr=2.98e-04 | gnorm=0.617 | tok/s=29,292 | MFU=7.0% | step=143189ms | tokens=1.316B | mem=4.2/32.7GB | cost=$40.18/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 12:05 (129.8h) | Phase3: SKIPPED | SFT: Oct 27 16:50 (134.5h) | DPO (Final): Oct 27 17:24 (135.1h) [2026-10-22 02:19:50][INFO][training.pretrain_engine] Starting global step 491 (accumulating 32 micro-batches)... [2026-10-22 02:19:50][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 02:20:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2665 [2026-10-22 02:20:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3350 [2026-10-22 02:20:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7901 [2026-10-22 02:20:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3912 [2026-10-22 02:20:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1145 [2026-10-22 02:20:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1164 [2026-10-22 02:20:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0039 [2026-10-22 02:20:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2217 [2026-10-22 02:20:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0187 [2026-10-22 02:20:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9127 [2026-10-22 02:20:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4916 [2026-10-22 02:20:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0532 [2026-10-22 02:21:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4236 [2026-10-22 02:21:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0317 [2026-10-22 02:21:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4068 [2026-10-22 02:21:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9963 [2026-10-22 02:21:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=3.6805 [2026-10-22 02:21:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9384 [2026-10-22 02:21:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0713 [2026-10-22 02:21:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8469 [2026-10-22 02:21:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.0558 [2026-10-22 02:21:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8679 [2026-10-22 02:21:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4416 [2026-10-22 02:21:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=3.9110 [2026-10-22 02:21:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0554 [2026-10-22 02:21:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9436 [2026-10-22 02:21:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7714 [2026-10-22 02:21:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0297 [2026-10-22 02:22:08][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,585 | MFU: 1.8% [2026-10-22 02:22:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2782 [2026-10-22 02:22:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2236 [2026-10-22 02:22:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5639 [2026-10-22 02:22:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0646 [2026-10-22 02:22:12][INFO][training.pretrain_engine] [Ph2|Step 491] loss=4.9823 | ppl=145.80 | aux=0.02505 | lr=2.98e-04 | gnorm=0.684 | tok/s=29,292 | MFU=7.0% | step=143188ms | tokens=1.317B | mem=4.2/32.7GB | cost=$40.26/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 12:07 (129.8h) | Phase3: SKIPPED | SFT: Oct 27 16:52 (134.5h) | DPO (Final): Oct 27 17:26 (135.1h) [2026-10-22 02:22:12][INFO][training.pretrain_engine] Starting global step 492 (accumulating 32 micro-batches)... [2026-10-22 02:22:12][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 02:22:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8717 [2026-10-22 02:22:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5226 [2026-10-22 02:22:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0559 [2026-10-22 02:22:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8393 [2026-10-22 02:22:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2032 [2026-10-22 02:22:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0467 [2026-10-22 02:22:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1084 [2026-10-22 02:22:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2262 [2026-10-22 02:23:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6963 [2026-10-22 02:23:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9680 [2026-10-22 02:23:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3012 [2026-10-22 02:23:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7216 [2026-10-22 02:23:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9249 [2026-10-22 02:23:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0093 [2026-10-22 02:23:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6541 [2026-10-22 02:23:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5754 [2026-10-22 02:23:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6721 [2026-10-22 02:23:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8931 [2026-10-22 02:23:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7073 [2026-10-22 02:23:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6442 [2026-10-22 02:23:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7730 [2026-10-22 02:24:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8150 [2026-10-22 02:24:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5767 [2026-10-22 02:24:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8594 [2026-10-22 02:24:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0177 [2026-10-22 02:24:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.1121 [2026-10-22 02:24:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2109 [2026-10-22 02:24:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0224 [2026-10-22 02:24:31][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,585 | MFU: 1.8% [2026-10-22 02:24:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6550 [2026-10-22 02:24:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1148 [2026-10-22 02:24:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9857 [2026-10-22 02:24:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1198 [2026-10-22 02:24:36][INFO][training.pretrain_engine] [Ph2|Step 492] loss=4.9848 | ppl=146.17 | aux=0.02507 | lr=2.98e-04 | gnorm=0.691 | tok/s=29,293 | MFU=7.0% | step=143187ms | tokens=1.318B | mem=4.2/32.7GB | cost=$40.34/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 12:09 (129.7h) | Phase3: SKIPPED | SFT: Oct 27 16:53 (134.5h) | DPO (Final): Oct 27 17:27 (135.1h) [2026-10-22 02:24:36][INFO][training.pretrain_engine] Starting global step 493 (accumulating 32 micro-batches)... [2026-10-22 02:24:36][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 02:24:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6721 [2026-10-22 02:24:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6212 [2026-10-22 02:24:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6573 [2026-10-22 02:24:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4941 [2026-10-22 02:25:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3252 [2026-10-22 02:25:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4940 [2026-10-22 02:25:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3411 [2026-10-22 02:25:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9979 [2026-10-22 02:25:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0002 [2026-10-22 02:25:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9410 [2026-10-22 02:25:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1702 [2026-10-22 02:25:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1235 [2026-10-22 02:25:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5412 [2026-10-22 02:25:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8668 [2026-10-22 02:25:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7085 [2026-10-22 02:25:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8463 [2026-10-22 02:26:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1163 [2026-10-22 02:26:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2472 [2026-10-22 02:26:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6017 [2026-10-22 02:26:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1290 [2026-10-22 02:26:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7491 [2026-10-22 02:26:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6714 [2026-10-22 02:26:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5637 [2026-10-22 02:26:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6363 [2026-10-22 02:26:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8060 [2026-10-22 02:26:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9503 [2026-10-22 02:26:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9512 [2026-10-22 02:26:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0839 [2026-10-22 02:26:54][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,572 | MFU: 1.8% [2026-10-22 02:26:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2637 [2026-10-22 02:26:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9060 [2026-10-22 02:26:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8556 [2026-10-22 02:26:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8563 [2026-10-22 02:26:59][INFO][training.pretrain_engine] [Ph2|Step 493] loss=4.8519 | ppl=127.98 | aux=0.02505 | lr=2.98e-04 | gnorm=0.547 | tok/s=29,293 | MFU=7.0% | step=143185ms | tokens=1.319B | mem=4.2/32.7GB | cost=$40.43/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 12:10 (129.7h) | Phase3: SKIPPED | SFT: Oct 27 16:55 (134.5h) | DPO (Final): Oct 27 17:29 (135.0h) [2026-10-22 02:26:59][INFO][training.pretrain_engine] Starting global step 494 (accumulating 32 micro-batches)... [2026-10-22 02:26:59][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 02:27:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5412 [2026-10-22 02:27:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9365 [2026-10-22 02:27:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4003 [2026-10-22 02:27:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6541 [2026-10-22 02:27:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6362 [2026-10-22 02:27:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0846 [2026-10-22 02:27:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4662 [2026-10-22 02:27:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8699 [2026-10-22 02:27:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6571 [2026-10-22 02:27:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6078 [2026-10-22 02:27:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3593 [2026-10-22 02:27:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1571 [2026-10-22 02:28:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0034 [2026-10-22 02:28:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0921 [2026-10-22 02:28:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0012 [2026-10-22 02:28:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1227 [2026-10-22 02:28:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7299 [2026-10-22 02:28:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5850 [2026-10-22 02:28:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3324 [2026-10-22 02:28:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6067 [2026-10-22 02:28:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0538 [2026-10-22 02:28:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6395 [2026-10-22 02:28:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3613 [2026-10-22 02:28:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9541 [2026-10-22 02:29:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5463 [2026-10-22 02:29:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8743 [2026-10-22 02:29:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9035 [2026-10-22 02:29:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9964 [2026-10-22 02:29:17][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,582 | MFU: 1.8% [2026-10-22 02:29:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0801 [2026-10-22 02:29:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4736 [2026-10-22 02:29:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1440 [2026-10-22 02:29:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0489 [2026-10-22 02:29:22][INFO][training.pretrain_engine] [Ph2|Step 494] loss=5.0526 | ppl=156.43 | aux=0.02505 | lr=2.98e-04 | gnorm=0.594 | tok/s=29,293 | MFU=7.0% | step=143185ms | tokens=1.320B | mem=4.2/32.7GB | cost=$40.51/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 12:12 (129.7h) | Phase3: SKIPPED | SFT: Oct 27 16:57 (134.5h) | DPO (Final): Oct 27 17:31 (135.0h) [2026-10-22 02:29:22][INFO][training.pretrain_engine] Starting global step 495 (accumulating 32 micro-batches)... [2026-10-22 02:29:22][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 02:29:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9387 [2026-10-22 02:29:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6469 [2026-10-22 02:29:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6726 [2026-10-22 02:29:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2725 [2026-10-22 02:29:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2177 [2026-10-22 02:29:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0024 [2026-10-22 02:29:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5619 [2026-10-22 02:29:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7486 [2026-10-22 02:30:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1982 [2026-10-22 02:30:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7588 [2026-10-22 02:30:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9502 [2026-10-22 02:30:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7324 [2026-10-22 02:30:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3734 [2026-10-22 02:30:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8489 [2026-10-22 02:30:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4897 [2026-10-22 02:30:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5583 [2026-10-22 02:30:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3620 [2026-10-22 02:30:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7249 [2026-10-22 02:30:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3009 [2026-10-22 02:30:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3701 [2026-10-22 02:31:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3094 [2026-10-22 02:31:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9995 [2026-10-22 02:31:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2458 [2026-10-22 02:31:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1022 [2026-10-22 02:31:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3583 [2026-10-22 02:31:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8189 [2026-10-22 02:31:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7190 [2026-10-22 02:31:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9416 [2026-10-22 02:31:41][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,574 | MFU: 1.8% [2026-10-22 02:31:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0565 [2026-10-22 02:31:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2738 [2026-10-22 02:31:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4233 [2026-10-22 02:31:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1273 [2026-10-22 02:31:46][INFO][training.pretrain_engine] [Ph2|Step 495] loss=4.8129 | ppl=123.09 | aux=0.02506 | lr=2.98e-04 | gnorm=0.699 | tok/s=29,291 | MFU=7.0% | step=143197ms | tokens=1.321B | mem=4.2/32.7GB | cost=$40.59/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 12:15 (129.7h) | Phase3: SKIPPED | SFT: Oct 27 16:59 (134.5h) | DPO (Final): Oct 27 17:33 (135.0h) [2026-10-22 02:31:49][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step495 | Tokens: 1.321B | Spent: $40.59 | Remaining: $959.41 [2026-10-22 02:31:49][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step495 [2026-10-22 02:31:49][INFO][training.pretrain_engine] Starting global step 496 (accumulating 32 micro-batches)... [2026-10-22 02:31:49][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 02:31:49][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-22 02:31:49][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-22 02:31:49][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-22 02:31:49][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step495 to Phase-Technologies/forge-500m-pretrain (path: phase2_step495)... [2026-10-22 02:31:49][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-22 02:31:58][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-22 02:31:58][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step495 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-22 02:32:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0385 [2026-10-22 02:32:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4197 [2026-10-22 02:32:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6057 [2026-10-22 02:32:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5117 [2026-10-22 02:32:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9228 [2026-10-22 02:32:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4276 [2026-10-22 02:32:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0824 [2026-10-22 02:32:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7739 [2026-10-22 02:32:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2200 [2026-10-22 02:32:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7483 [2026-10-22 02:32:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6658 [2026-10-22 02:32:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4176 [2026-10-22 02:32:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8190 [2026-10-22 02:32:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0700 [2026-10-22 02:32:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0938 [2026-10-22 02:33:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4138 [2026-10-22 02:33:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4533 [2026-10-22 02:33:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0882 [2026-10-22 02:33:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0225 [2026-10-22 02:33:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0808 [2026-10-22 02:33:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2691 [2026-10-22 02:33:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3212 [2026-10-22 02:33:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5595 [2026-10-22 02:33:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4112 [2026-10-22 02:33:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5882 [2026-10-22 02:33:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7561 [2026-10-22 02:33:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9525 [2026-10-22 02:33:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7097 [2026-10-22 02:34:07][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,604 | MFU: 1.8% [2026-10-22 02:34:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0606 [2026-10-22 02:34:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0432 [2026-10-22 02:34:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6889 [2026-10-22 02:34:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9047 [2026-10-22 02:34:11][INFO][training.pretrain_engine] [Ph2|Step 496] loss=4.9205 | ppl=137.06 | aux=0.02505 | lr=2.98e-04 | gnorm=0.566 | tok/s=29,291 | MFU=7.0% | step=143193ms | tokens=1.322B | mem=4.2/32.7GB | cost=$40.67/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 12:16 (129.7h) | Phase3: SKIPPED | SFT: Oct 27 17:01 (134.5h) | DPO (Final): Oct 27 17:35 (135.0h) [2026-10-22 02:34:11][INFO][training.pretrain_engine] Starting global step 497 (accumulating 32 micro-batches)... [2026-10-22 02:34:11][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 02:34:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2106 [2026-10-22 02:34:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3390 [2026-10-22 02:34:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2674 [2026-10-22 02:34:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3309 [2026-10-22 02:34:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4261 [2026-10-22 02:34:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0273 [2026-10-22 02:34:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2937 [2026-10-22 02:34:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1909 [2026-10-22 02:35:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0572 [2026-10-22 02:35:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2877 [2026-10-22 02:35:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1869 [2026-10-22 02:35:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2932 [2026-10-22 02:35:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1676 [2026-10-22 02:35:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5906 [2026-10-22 02:35:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7136 [2026-10-22 02:35:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4948 [2026-10-22 02:35:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8945 [2026-10-22 02:35:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9038 [2026-10-22 02:35:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6148 [2026-10-22 02:35:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6538 [2026-10-22 02:35:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3790 [2026-10-22 02:35:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8958 [2026-10-22 02:35:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8821 [2026-10-22 02:35:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6861 [2026-10-22 02:36:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7188 [2026-10-22 02:36:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8567 [2026-10-22 02:36:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2573 [2026-10-22 02:36:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6360 [2026-10-22 02:36:29][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,593 | MFU: 1.8% [2026-10-22 02:36:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9244 [2026-10-22 02:36:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8657 [2026-10-22 02:36:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7976 [2026-10-22 02:36:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8136 [2026-10-22 02:36:34][INFO][training.pretrain_engine] [Ph2|Step 497] loss=4.9883 | ppl=146.69 | aux=0.02506 | lr=2.98e-04 | gnorm=0.547 | tok/s=29,289 | MFU=7.0% | step=143203ms | tokens=1.323B | mem=4.2/32.7GB | cost=$40.76/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 12:19 (129.7h) | Phase3: SKIPPED | SFT: Oct 27 17:03 (134.5h) | DPO (Final): Oct 27 17:37 (135.0h) [2026-10-22 02:36:34][INFO][training.pretrain_engine] Starting global step 498 (accumulating 32 micro-batches)... [2026-10-22 02:36:34][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 02:36:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9774 [2026-10-22 02:36:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4547 [2026-10-22 02:36:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7841 [2026-10-22 02:36:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8326 [2026-10-22 02:37:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6114 [2026-10-22 02:37:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1859 [2026-10-22 02:37:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2418 [2026-10-22 02:37:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2434 [2026-10-22 02:37:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9029 [2026-10-22 02:37:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8771 [2026-10-22 02:37:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5136 [2026-10-22 02:37:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7708 [2026-10-22 02:37:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1737 [2026-10-22 02:37:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1344 [2026-10-22 02:37:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1688 [2026-10-22 02:37:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2962 [2026-10-22 02:38:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9050 [2026-10-22 02:38:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3269 [2026-10-22 02:38:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8673 [2026-10-22 02:38:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8333 [2026-10-22 02:38:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1028 [2026-10-22 02:38:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7637 [2026-10-22 02:38:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8159 [2026-10-22 02:38:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4488 [2026-10-22 02:38:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8061 [2026-10-22 02:38:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7960 [2026-10-22 02:38:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4894 [2026-10-22 02:38:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4952 [2026-10-22 02:38:52][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,598 | MFU: 1.8% [2026-10-22 02:38:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0797 [2026-10-22 02:38:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8558 [2026-10-22 02:38:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9895 [2026-10-22 02:38:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7018 [2026-10-22 02:38:58][INFO][training.pretrain_engine] [Ph2|Step 498] loss=4.9003 | ppl=134.33 | aux=0.02506 | lr=2.98e-04 | gnorm=0.520 | tok/s=29,289 | MFU=7.0% | step=143204ms | tokens=1.324B | mem=4.2/32.7GB | cost=$40.84/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 12:21 (129.7h) | Phase3: SKIPPED | SFT: Oct 27 17:05 (134.4h) | DPO (Final): Oct 27 17:39 (135.0h) [2026-10-22 02:38:58][INFO][training.pretrain_engine] Starting global step 499 (accumulating 32 micro-batches)... [2026-10-22 02:38:58][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 02:39:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0524 [2026-10-22 02:39:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1644 [2026-10-22 02:39:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6398 [2026-10-22 02:39:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6364 [2026-10-22 02:39:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3246 [2026-10-22 02:39:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4481 [2026-10-22 02:39:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5362 [2026-10-22 02:39:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8202 [2026-10-22 02:39:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2596 [2026-10-22 02:39:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2841 [2026-10-22 02:39:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4028 [2026-10-22 02:39:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5339 [2026-10-22 02:40:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0536 [2026-10-22 02:40:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1130 [2026-10-22 02:40:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9814 [2026-10-22 02:40:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7918 [2026-10-22 02:40:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9979 [2026-10-22 02:40:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1109 [2026-10-22 02:40:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0274 [2026-10-22 02:40:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5136 [2026-10-22 02:40:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3697 [2026-10-22 02:40:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1862 [2026-10-22 02:40:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3511 [2026-10-22 02:40:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1218 [2026-10-22 02:41:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8692 [2026-10-22 02:41:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7773 [2026-10-22 02:41:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8415 [2026-10-22 02:41:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5891 [2026-10-22 02:41:15][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,603 | MFU: 1.8% [2026-10-22 02:41:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9497 [2026-10-22 02:41:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9288 [2026-10-22 02:41:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4676 [2026-10-22 02:41:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5926 [2026-10-22 02:41:21][INFO][training.pretrain_engine] [Ph2|Step 499] loss=4.9780 | ppl=145.19 | aux=0.02506 | lr=2.98e-04 | gnorm=0.520 | tok/s=29,289 | MFU=7.0% | step=143205ms | tokens=1.325B | mem=4.2/32.7GB | cost=$40.92/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 12:22 (129.7h) | Phase3: SKIPPED | SFT: Oct 27 17:07 (134.4h) | DPO (Final): Oct 27 17:41 (135.0h) [2026-10-22 02:41:21][INFO][training.pretrain_engine] Starting global step 500 (accumulating 32 micro-batches)... [2026-10-22 02:41:21][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 02:41:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1595 [2026-10-22 02:41:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8009 [2026-10-22 02:41:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0082 [2026-10-22 02:41:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6487 [2026-10-22 02:41:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9898 [2026-10-22 02:41:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8263 [2026-10-22 02:41:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6529 [2026-10-22 02:41:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2747 [2026-10-22 02:42:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2623 [2026-10-22 02:42:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8206 [2026-10-22 02:42:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8708 [2026-10-22 02:42:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5926 [2026-10-22 02:42:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7429 [2026-10-22 02:42:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3417 [2026-10-22 02:42:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1336 [2026-10-22 02:42:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3468 [2026-10-22 02:42:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8306 [2026-10-22 02:42:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3339 [2026-10-22 02:42:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8149 [2026-10-22 02:42:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4899 [2026-10-22 02:43:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9842 [2026-10-22 02:43:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9751 [2026-10-22 02:43:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0033 [2026-10-22 02:43:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8472 [2026-10-22 02:43:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8169 [2026-10-22 02:43:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3948 [2026-10-22 02:43:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7906 [2026-10-22 02:43:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8254 [2026-10-22 02:43:39][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,606 | MFU: 1.8% [2026-10-22 02:43:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1163 [2026-10-22 02:43:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1005 [2026-10-22 02:43:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0108 [2026-10-22 02:43:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7029 [2026-10-22 02:43:44][INFO][training.pretrain_engine] [Ph2|Step 500] loss=4.9393 | ppl=139.67 | aux=0.02506 | lr=2.98e-04 | gnorm=0.471 | tok/s=29,288 | MFU=7.0% | step=143207ms | tokens=1.326B | mem=4.2/32.7GB | cost=$41.00/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 12:24 (129.7h) | Phase3: SKIPPED | SFT: Oct 27 17:09 (134.4h) | DPO (Final): Oct 27 17:43 (135.0h) [2026-10-22 02:43:44][INFO][training.pretrain_engine] Starting global step 501 (accumulating 32 micro-batches)... [2026-10-22 02:43:44][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 02:44:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4202 [2026-10-22 02:44:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9177 [2026-10-22 02:44:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1366 [2026-10-22 02:44:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6055 [2026-10-22 02:44:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1609 [2026-10-22 02:44:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1410 [2026-10-22 02:44:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5280 [2026-10-22 02:44:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4317 [2026-10-22 02:44:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0998 [2026-10-22 02:44:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3830 [2026-10-22 02:44:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9556 [2026-10-22 02:44:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0402 [2026-10-22 02:44:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7598 [2026-10-22 02:44:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7182 [2026-10-22 02:44:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8280 [2026-10-22 02:44:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4588 [2026-10-22 02:45:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0958 [2026-10-22 02:45:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.2436 [2026-10-22 02:45:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=3.9408 [2026-10-22 02:45:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4182 [2026-10-22 02:45:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0581 [2026-10-22 02:45:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9482 [2026-10-22 02:45:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1930 [2026-10-22 02:45:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0895 [2026-10-22 02:45:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3644 [2026-10-22 02:45:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1314 [2026-10-22 02:45:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7392 [2026-10-22 02:45:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7371 [2026-10-22 02:46:02][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,606 | MFU: 1.8% [2026-10-22 02:46:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8358 [2026-10-22 02:46:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3537 [2026-10-22 02:46:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1527 [2026-10-22 02:46:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8879 [2026-10-22 02:46:07][INFO][training.pretrain_engine] [Ph2|Step 501] loss=4.9719 | ppl=144.31 | aux=0.02507 | lr=2.98e-04 | gnorm=0.438 | tok/s=29,288 | MFU=7.0% | step=143210ms | tokens=1.327B | mem=4.2/32.7GB | cost=$41.08/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 12:26 (129.7h) | Phase3: SKIPPED | SFT: Oct 27 17:11 (134.4h) | DPO (Final): Oct 27 17:45 (135.0h) [2026-10-22 02:46:07][INFO][training.pretrain_engine] Starting global step 502 (accumulating 32 micro-batches)... [2026-10-22 02:46:07][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 02:46:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5631 [2026-10-22 02:46:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3978 [2026-10-22 02:46:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3630 [2026-10-22 02:46:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4798 [2026-10-22 02:46:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8760 [2026-10-22 02:46:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7229 [2026-10-22 02:46:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9381 [2026-10-22 02:46:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0498 [2026-10-22 02:47:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3505 [2026-10-22 02:47:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0906 [2026-10-22 02:47:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2847 [2026-10-22 02:47:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1571 [2026-10-22 02:47:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1135 [2026-10-22 02:47:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9464 [2026-10-22 02:47:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3087 [2026-10-22 02:47:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5256 [2026-10-22 02:47:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0690 [2026-10-22 02:47:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9801 [2026-10-22 02:47:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6015 [2026-10-22 02:47:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9477 [2026-10-22 02:47:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8612 [2026-10-22 02:47:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1555 [2026-10-22 02:47:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1795 [2026-10-22 02:47:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0104 [2026-10-22 02:48:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8601 [2026-10-22 02:48:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3282 [2026-10-22 02:48:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7012 [2026-10-22 02:48:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8120 [2026-10-22 02:48:26][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,584 | MFU: 1.8% [2026-10-22 02:48:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1447 [2026-10-22 02:48:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0854 [2026-10-22 02:48:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2510 [2026-10-22 02:48:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5868 [2026-10-22 02:48:30][INFO][training.pretrain_engine] [Ph2|Step 502] loss=4.9386 | ppl=139.57 | aux=0.02506 | lr=2.98e-04 | gnorm=0.520 | tok/s=29,289 | MFU=7.0% | step=143206ms | tokens=1.329B | mem=4.2/32.7GB | cost=$41.17/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 12:28 (129.7h) | Phase3: SKIPPED | SFT: Oct 27 17:12 (134.4h) | DPO (Final): Oct 27 17:46 (135.0h) [2026-10-22 02:48:30][INFO][training.pretrain_engine] Starting global step 503 (accumulating 32 micro-batches)... [2026-10-22 02:48:30][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 02:48:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5613 [2026-10-22 02:48:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8948 [2026-10-22 02:48:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9752 [2026-10-22 02:48:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6918 [2026-10-22 02:49:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7063 [2026-10-22 02:49:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5427 [2026-10-22 02:49:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0072 [2026-10-22 02:49:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3061 [2026-10-22 02:49:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6517 [2026-10-22 02:49:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8785 [2026-10-22 02:49:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0392 [2026-10-22 02:49:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7694 [2026-10-22 02:49:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7744 [2026-10-22 02:49:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2632 [2026-10-22 02:49:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7095 [2026-10-22 02:49:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8061 [2026-10-22 02:49:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0412 [2026-10-22 02:49:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9844 [2026-10-22 02:50:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2644 [2026-10-22 02:50:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0481 [2026-10-22 02:50:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0413 [2026-10-22 02:50:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1704 [2026-10-22 02:50:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4157 [2026-10-22 02:50:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1119 [2026-10-22 02:50:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4188 [2026-10-22 02:50:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4806 [2026-10-22 02:50:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6559 [2026-10-22 02:50:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=3.9277 [2026-10-22 02:50:49][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,587 | MFU: 1.8% [2026-10-22 02:50:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4216 [2026-10-22 02:50:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4568 [2026-10-22 02:50:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.0143 [2026-10-22 02:50:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8830 [2026-10-22 02:50:54][INFO][training.pretrain_engine] [Ph2|Step 503] loss=4.9966 | ppl=147.92 | aux=0.02505 | lr=2.98e-04 | gnorm=0.602 | tok/s=29,288 | MFU=7.0% | step=143211ms | tokens=1.330B | mem=4.2/32.7GB | cost=$41.25/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 12:30 (129.7h) | Phase3: SKIPPED | SFT: Oct 27 17:14 (134.4h) | DPO (Final): Oct 27 17:48 (135.0h) [2026-10-22 02:50:54][INFO][training.pretrain_engine] Starting global step 504 (accumulating 32 micro-batches)... [2026-10-22 02:50:54][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 02:51:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4421 [2026-10-22 02:51:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1881 [2026-10-22 02:51:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8539 [2026-10-22 02:51:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7696 [2026-10-22 02:51:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9855 [2026-10-22 02:51:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8532 [2026-10-22 02:51:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6443 [2026-10-22 02:51:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9488 [2026-10-22 02:51:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3503 [2026-10-22 02:51:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6700 [2026-10-22 02:51:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0407 [2026-10-22 02:51:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2356 [2026-10-22 02:52:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6329 [2026-10-22 02:52:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7907 [2026-10-22 02:52:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9616 [2026-10-22 02:52:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6288 [2026-10-22 02:52:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2010 [2026-10-22 02:52:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1599 [2026-10-22 02:52:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9820 [2026-10-22 02:52:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1409 [2026-10-22 02:52:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4693 [2026-10-22 02:52:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0743 [2026-10-22 02:52:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2550 [2026-10-22 02:52:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9016 [2026-10-22 02:52:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9396 [2026-10-22 02:52:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.1119 [2026-10-22 02:52:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8860 [2026-10-22 02:52:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8428 [2026-10-22 02:53:12][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,589 | MFU: 1.8% [2026-10-22 02:53:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9464 [2026-10-22 02:53:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5457 [2026-10-22 02:53:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8878 [2026-10-22 02:53:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7212 [2026-10-22 02:53:17][INFO][training.pretrain_engine] [Ph2|Step 504] loss=5.0122 | ppl=150.23 | aux=0.02505 | lr=2.98e-04 | gnorm=0.723 | tok/s=29,287 | MFU=7.0% | step=143216ms | tokens=1.331B | mem=4.2/32.7GB | cost=$41.33/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 12:32 (129.7h) | Phase3: SKIPPED | SFT: Oct 27 17:16 (134.4h) | DPO (Final): Oct 27 17:51 (135.0h) [2026-10-22 02:53:20][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step504 | Tokens: 1.331B | Spent: $41.33 | Remaining: $958.67 [2026-10-22 02:53:20][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step504 [2026-10-22 02:53:20][INFO][training.pretrain_engine] Starting global step 505 (accumulating 32 micro-batches)... [2026-10-22 02:53:20][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 02:53:20][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-22 02:53:20][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-22 02:53:20][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-22 02:53:20][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step504 to Phase-Technologies/forge-500m-pretrain (path: phase2_step504)... [2026-10-22 02:53:21][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-22 02:53:30][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-22 02:53:30][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step504 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-22 02:53:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9664 [2026-10-22 02:53:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7320 [2026-10-22 02:53:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9429 [2026-10-22 02:53:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6263 [2026-10-22 02:53:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.0216 [2026-10-22 02:53:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5715 [2026-10-22 02:53:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2738 [2026-10-22 02:53:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1554 [2026-10-22 02:54:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8101 [2026-10-22 02:54:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6640 [2026-10-22 02:54:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6035 [2026-10-22 02:54:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5677 [2026-10-22 02:54:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5127 [2026-10-22 02:54:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4843 [2026-10-22 02:54:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2416 [2026-10-22 02:54:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=3.9057 [2026-10-22 02:54:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0502 [2026-10-22 02:54:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7027 [2026-10-22 02:54:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9267 [2026-10-22 02:54:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3382 [2026-10-22 02:55:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1413 [2026-10-22 02:55:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1239 [2026-10-22 02:55:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.1059 [2026-10-22 02:55:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.1146 [2026-10-22 02:55:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0583 [2026-10-22 02:55:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3519 [2026-10-22 02:55:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0458 [2026-10-22 02:55:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6554 [2026-10-22 02:55:38][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,594 | MFU: 1.8% [2026-10-22 02:55:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.1705 [2026-10-22 02:55:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8916 [2026-10-22 02:55:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6577 [2026-10-22 02:55:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6576 [2026-10-22 02:55:43][INFO][training.pretrain_engine] [Ph2|Step 505] loss=4.8610 | ppl=129.15 | aux=0.02506 | lr=2.98e-04 | gnorm=0.844 | tok/s=29,287 | MFU=7.0% | step=143215ms | tokens=1.332B | mem=4.2/32.7GB | cost=$41.41/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 12:34 (129.6h) | Phase3: SKIPPED | SFT: Oct 27 17:18 (134.4h) | DPO (Final): Oct 27 17:52 (135.0h) [2026-10-22 02:55:43][INFO][training.pretrain_engine] Starting global step 506 (accumulating 32 micro-batches)... [2026-10-22 02:55:43][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 02:56:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2713 [2026-10-22 02:56:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7343 [2026-10-22 02:56:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9623 [2026-10-22 02:56:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2266 [2026-10-22 02:56:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4943 [2026-10-22 02:56:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.0258 [2026-10-22 02:56:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5576 [2026-10-22 02:56:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8146 [2026-10-22 02:56:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4977 [2026-10-22 02:56:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=3.7222 [2026-10-22 02:56:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2281 [2026-10-22 02:56:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.1284 [2026-10-22 02:56:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0458 [2026-10-22 02:56:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2879 [2026-10-22 02:56:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3006 [2026-10-22 02:56:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0369 [2026-10-22 02:57:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0149 [2026-10-22 02:57:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1002 [2026-10-22 02:57:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5865 [2026-10-22 02:57:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2136 [2026-10-22 02:57:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9663 [2026-10-22 02:57:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6693 [2026-10-22 02:57:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4670 [2026-10-22 02:57:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6467 [2026-10-22 02:57:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0415 [2026-10-22 02:57:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0524 [2026-10-22 02:57:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9994 [2026-10-22 02:57:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5104 [2026-10-22 02:58:01][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,577 | MFU: 1.8% [2026-10-22 02:58:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=3.8635 [2026-10-22 02:58:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=3.8896 [2026-10-22 02:58:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2032 [2026-10-22 02:58:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=3.5892 [2026-10-22 02:58:06][INFO][training.pretrain_engine] [Ph2|Step 506] loss=4.9087 | ppl=135.47 | aux=0.02506 | lr=2.98e-04 | gnorm=0.930 | tok/s=29,285 | MFU=7.0% | step=143222ms | tokens=1.333B | mem=4.2/32.7GB | cost=$41.49/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 12:36 (129.6h) | Phase3: SKIPPED | SFT: Oct 27 17:20 (134.4h) | DPO (Final): Oct 27 17:55 (134.9h) [2026-10-22 02:58:06][INFO][training.pretrain_engine] Starting global step 507 (accumulating 32 micro-batches)... [2026-10-22 02:58:06][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 02:58:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8038 [2026-10-22 02:58:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9893 [2026-10-22 02:58:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5252 [2026-10-22 02:58:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0799 [2026-10-22 02:58:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9243 [2026-10-22 02:58:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9650 [2026-10-22 02:58:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1414 [2026-10-22 02:58:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9840 [2026-10-22 02:59:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3441 [2026-10-22 02:59:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8046 [2026-10-22 02:59:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8252 [2026-10-22 02:59:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7366 [2026-10-22 02:59:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7403 [2026-10-22 02:59:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1459 [2026-10-22 02:59:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2175 [2026-10-22 02:59:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9600 [2026-10-22 02:59:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5175 [2026-10-22 02:59:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2371 [2026-10-22 02:59:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6431 [2026-10-22 02:59:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8537 [2026-10-22 02:59:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8535 [2026-10-22 02:59:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7476 [2026-10-22 02:59:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1058 [2026-10-22 02:59:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7928 [2026-10-22 03:00:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2339 [2026-10-22 03:00:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0877 [2026-10-22 03:00:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9989 [2026-10-22 03:00:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3181 [2026-10-22 03:00:25][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,586 | MFU: 1.8% [2026-10-22 03:00:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5735 [2026-10-22 03:00:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.0295 [2026-10-22 03:00:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0867 [2026-10-22 03:00:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9607 [2026-10-22 03:00:30][INFO][training.pretrain_engine] [Ph2|Step 507] loss=4.9224 | ppl=137.34 | aux=0.02508 | lr=2.98e-04 | gnorm=1.180 | tok/s=29,284 | MFU=7.0% | step=143230ms | tokens=1.334B | mem=4.2/32.7GB | cost=$41.58/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 12:38 (129.6h) | Phase3: SKIPPED | SFT: Oct 27 17:23 (134.4h) | DPO (Final): Oct 27 17:57 (134.9h) [2026-10-22 03:00:30][INFO][training.pretrain_engine] Starting global step 508 (accumulating 32 micro-batches)... [2026-10-22 03:00:30][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 03:00:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2990 [2026-10-22 03:00:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6064 [2026-10-22 03:00:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2318 [2026-10-22 03:00:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0359 [2026-10-22 03:01:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3277 [2026-10-22 03:01:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8242 [2026-10-22 03:01:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8813 [2026-10-22 03:01:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9631 [2026-10-22 03:01:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5632 [2026-10-22 03:01:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6300 [2026-10-22 03:01:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7554 [2026-10-22 03:01:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3551 [2026-10-22 03:01:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6311 [2026-10-22 03:01:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4605 [2026-10-22 03:01:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3034 [2026-10-22 03:01:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6060 [2026-10-22 03:01:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5265 [2026-10-22 03:01:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8478 [2026-10-22 03:01:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3280 [2026-10-22 03:02:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1277 [2026-10-22 03:02:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1725 [2026-10-22 03:02:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4597 [2026-10-22 03:02:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9931 [2026-10-22 03:02:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0677 [2026-10-22 03:02:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3011 [2026-10-22 03:02:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2605 [2026-10-22 03:02:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2654 [2026-10-22 03:02:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6963 [2026-10-22 03:02:48][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,577 | MFU: 1.8% [2026-10-22 03:02:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4054 [2026-10-22 03:02:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3852 [2026-10-22 03:02:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0894 [2026-10-22 03:02:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1292 [2026-10-22 03:02:53][INFO][training.pretrain_engine] [Ph2|Step 508] loss=4.9713 | ppl=144.22 | aux=0.02506 | lr=2.98e-04 | gnorm=0.969 | tok/s=29,284 | MFU=7.0% | step=143231ms | tokens=1.335B | mem=4.2/32.7GB | cost=$41.66/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 12:40 (129.6h) | Phase3: SKIPPED | SFT: Oct 27 17:24 (134.4h) | DPO (Final): Oct 27 17:59 (134.9h) [2026-10-22 03:02:53][INFO][training.pretrain_engine] Starting global step 509 (accumulating 32 micro-batches)... [2026-10-22 03:02:53][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 03:03:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5453 [2026-10-22 03:03:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0066 [2026-10-22 03:03:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0471 [2026-10-22 03:03:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0649 [2026-10-22 03:03:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4819 [2026-10-22 03:03:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8691 [2026-10-22 03:03:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2301 [2026-10-22 03:03:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1099 [2026-10-22 03:03:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7416 [2026-10-22 03:03:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7221 [2026-10-22 03:03:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9226 [2026-10-22 03:03:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7397 [2026-10-22 03:04:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8911 [2026-10-22 03:04:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9686 [2026-10-22 03:04:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8380 [2026-10-22 03:04:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1095 [2026-10-22 03:04:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9340 [2026-10-22 03:04:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1641 [2026-10-22 03:04:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0146 [2026-10-22 03:04:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9336 [2026-10-22 03:04:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9056 [2026-10-22 03:04:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1120 [2026-10-22 03:04:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8690 [2026-10-22 03:04:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8013 [2026-10-22 03:04:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0471 [2026-10-22 03:04:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0787 [2026-10-22 03:04:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0053 [2026-10-22 03:04:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9878 [2026-10-22 03:05:11][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,601 | MFU: 1.8% [2026-10-22 03:05:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0188 [2026-10-22 03:05:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1247 [2026-10-22 03:05:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3498 [2026-10-22 03:05:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0387 [2026-10-22 03:05:17][INFO][training.pretrain_engine] [Ph2|Step 509] loss=5.0541 | ppl=156.66 | aux=0.02506 | lr=2.98e-04 | gnorm=1.031 | tok/s=29,282 | MFU=7.0% | step=143239ms | tokens=1.336B | mem=4.2/32.7GB | cost=$41.74/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 12:42 (129.6h) | Phase3: SKIPPED | SFT: Oct 27 17:27 (134.4h) | DPO (Final): Oct 27 18:01 (134.9h) [2026-10-22 03:05:17][INFO][training.pretrain_engine] Starting global step 510 (accumulating 32 micro-batches)... [2026-10-22 03:05:17][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 03:05:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.0313 [2026-10-22 03:05:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5094 [2026-10-22 03:05:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4449 [2026-10-22 03:05:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3875 [2026-10-22 03:05:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8074 [2026-10-22 03:05:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7141 [2026-10-22 03:05:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8183 [2026-10-22 03:05:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1901 [2026-10-22 03:06:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9846 [2026-10-22 03:06:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1993 [2026-10-22 03:06:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0166 [2026-10-22 03:06:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2007 [2026-10-22 03:06:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1730 [2026-10-22 03:06:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3928 [2026-10-22 03:06:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3015 [2026-10-22 03:06:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2381 [2026-10-22 03:06:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.2900 [2026-10-22 03:06:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7305 [2026-10-22 03:06:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1165 [2026-10-22 03:06:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7167 [2026-10-22 03:07:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7790 [2026-10-22 03:07:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0481 [2026-10-22 03:07:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5766 [2026-10-22 03:07:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7183 [2026-10-22 03:07:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6147 [2026-10-22 03:07:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9920 [2026-10-22 03:07:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0536 [2026-10-22 03:07:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0540 [2026-10-22 03:07:35][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,620 | MFU: 1.8% [2026-10-22 03:07:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9545 [2026-10-22 03:07:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7572 [2026-10-22 03:07:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1448 [2026-10-22 03:07:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7391 [2026-10-22 03:07:40][INFO][training.pretrain_engine] [Ph2|Step 510] loss=4.8904 | ppl=133.01 | aux=0.02506 | lr=2.98e-04 | gnorm=0.691 | tok/s=29,281 | MFU=7.0% | step=143244ms | tokens=1.337B | mem=4.2/32.7GB | cost=$41.82/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 12:44 (129.6h) | Phase3: SKIPPED | SFT: Oct 27 17:29 (134.4h) | DPO (Final): Oct 27 18:03 (134.9h) [2026-10-22 03:07:40][INFO][training.pretrain_engine] Starting global step 511 (accumulating 32 micro-batches)... [2026-10-22 03:07:40][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 03:07:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3472 [2026-10-22 03:07:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2903 [2026-10-22 03:07:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9846 [2026-10-22 03:07:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3957 [2026-10-22 03:08:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1087 [2026-10-22 03:08:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8113 [2026-10-22 03:08:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8370 [2026-10-22 03:08:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3896 [2026-10-22 03:08:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6755 [2026-10-22 03:08:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5178 [2026-10-22 03:08:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2306 [2026-10-22 03:08:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1394 [2026-10-22 03:08:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2845 [2026-10-22 03:08:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1969 [2026-10-22 03:08:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9502 [2026-10-22 03:08:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1233 [2026-10-22 03:09:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1533 [2026-10-22 03:09:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7216 [2026-10-22 03:09:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4494 [2026-10-22 03:09:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4834 [2026-10-22 03:09:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2763 [2026-10-22 03:09:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3819 [2026-10-22 03:09:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0309 [2026-10-22 03:09:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.1022 [2026-10-22 03:09:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0902 [2026-10-22 03:09:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9025 [2026-10-22 03:09:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8942 [2026-10-22 03:09:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0224 [2026-10-22 03:09:58][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,608 | MFU: 1.8% [2026-10-22 03:10:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9094 [2026-10-22 03:10:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9924 [2026-10-22 03:10:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0145 [2026-10-22 03:10:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0559 [2026-10-22 03:10:04][INFO][training.pretrain_engine] [Ph2|Step 511] loss=5.0543 | ppl=156.70 | aux=0.02506 | lr=2.98e-04 | gnorm=0.812 | tok/s=29,280 | MFU=7.0% | step=143247ms | tokens=1.338B | mem=4.2/32.7GB | cost=$41.91/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 12:46 (129.6h) | Phase3: SKIPPED | SFT: Oct 27 17:31 (134.4h) | DPO (Final): Oct 27 18:05 (134.9h) [2026-10-22 03:10:04][INFO][training.pretrain_engine] Starting global step 512 (accumulating 32 micro-batches)... [2026-10-22 03:10:04][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 03:10:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3019 [2026-10-22 03:10:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3446 [2026-10-22 03:10:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0011 [2026-10-22 03:10:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7827 [2026-10-22 03:10:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0784 [2026-10-22 03:10:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2866 [2026-10-22 03:10:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8864 [2026-10-22 03:10:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8358 [2026-10-22 03:10:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8861 [2026-10-22 03:10:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5700 [2026-10-22 03:10:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2689 [2026-10-22 03:10:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9562 [2026-10-22 03:11:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8456 [2026-10-22 03:11:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9832 [2026-10-22 03:11:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9010 [2026-10-22 03:11:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9888 [2026-10-22 03:11:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3620 [2026-10-22 03:11:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0563 [2026-10-22 03:11:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9204 [2026-10-22 03:11:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0084 [2026-10-22 03:11:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6107 [2026-10-22 03:11:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7361 [2026-10-22 03:11:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0658 [2026-10-22 03:11:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6292 [2026-10-22 03:12:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1470 [2026-10-22 03:12:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6067 [2026-10-22 03:12:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3148 [2026-10-22 03:12:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0570 [2026-10-22 03:12:22][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,598 | MFU: 1.8% [2026-10-22 03:12:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6153 [2026-10-22 03:12:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2479 [2026-10-22 03:12:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3259 [2026-10-22 03:12:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0241 [2026-10-22 03:12:27][INFO][training.pretrain_engine] [Ph2|Step 512] loss=4.9155 | ppl=136.38 | aux=0.02507 | lr=2.98e-04 | gnorm=0.617 | tok/s=29,280 | MFU=7.0% | step=143250ms | tokens=1.339B | mem=4.2/32.7GB | cost=$41.99/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 12:48 (129.6h) | Phase3: SKIPPED | SFT: Oct 27 17:33 (134.3h) | DPO (Final): Oct 27 18:07 (134.9h) [2026-10-22 03:12:27][INFO][training.pretrain_engine] Starting global step 513 (accumulating 32 micro-batches)... [2026-10-22 03:12:27][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 03:12:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.6333 [2026-10-22 03:12:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4694 [2026-10-22 03:12:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3248 [2026-10-22 03:12:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6445 [2026-10-22 03:13:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6163 [2026-10-22 03:13:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2929 [2026-10-22 03:13:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2207 [2026-10-22 03:13:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0329 [2026-10-22 03:13:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1543 [2026-10-22 03:13:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8056 [2026-10-22 03:13:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3198 [2026-10-22 03:13:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7631 [2026-10-22 03:13:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9966 [2026-10-22 03:13:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9176 [2026-10-22 03:13:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1307 [2026-10-22 03:13:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0346 [2026-10-22 03:13:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1244 [2026-10-22 03:13:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9288 [2026-10-22 03:13:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8945 [2026-10-22 03:13:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3573 [2026-10-22 03:14:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1215 [2026-10-22 03:14:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1344 [2026-10-22 03:14:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7941 [2026-10-22 03:14:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3239 [2026-10-22 03:14:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2706 [2026-10-22 03:14:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9581 [2026-10-22 03:14:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7419 [2026-10-22 03:14:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9404 [2026-10-22 03:14:45][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,582 | MFU: 1.8% [2026-10-22 03:14:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4091 [2026-10-22 03:14:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5803 [2026-10-22 03:14:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=3.8723 [2026-10-22 03:14:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7073 [2026-10-22 03:14:50][INFO][training.pretrain_engine] [Ph2|Step 513] loss=4.9311 | ppl=138.53 | aux=0.02507 | lr=2.98e-04 | gnorm=0.734 | tok/s=29,279 | MFU=7.0% | step=143255ms | tokens=1.340B | mem=4.2/32.7GB | cost=$42.07/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 12:50 (129.6h) | Phase3: SKIPPED | SFT: Oct 27 17:35 (134.3h) | DPO (Final): Oct 27 18:09 (134.9h) [2026-10-22 03:14:53][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step513 | Tokens: 1.340B | Spent: $42.07 | Remaining: $957.93 [2026-10-22 03:14:53][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step513 [2026-10-22 03:14:54][INFO][training.pretrain_engine] Starting global step 514 (accumulating 32 micro-batches)... [2026-10-22 03:14:54][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 03:14:54][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-22 03:14:54][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-22 03:14:54][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-22 03:14:54][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step513 to Phase-Technologies/forge-500m-pretrain (path: phase2_step513)... [2026-10-22 03:14:54][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-22 03:15:04][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-22 03:15:04][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step513 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-22 03:15:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0045 [2026-10-22 03:15:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6732 [2026-10-22 03:15:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.1313 [2026-10-22 03:15:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5296 [2026-10-22 03:15:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1241 [2026-10-22 03:15:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1301 [2026-10-22 03:15:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0353 [2026-10-22 03:15:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0739 [2026-10-22 03:15:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2040 [2026-10-22 03:15:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8954 [2026-10-22 03:15:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9831 [2026-10-22 03:15:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9622 [2026-10-22 03:16:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6211 [2026-10-22 03:16:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8675 [2026-10-22 03:16:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6942 [2026-10-22 03:16:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4621 [2026-10-22 03:16:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3183 [2026-10-22 03:16:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0077 [2026-10-22 03:16:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3492 [2026-10-22 03:16:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1442 [2026-10-22 03:16:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4356 [2026-10-22 03:16:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2595 [2026-10-22 03:16:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1783 [2026-10-22 03:16:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7426 [2026-10-22 03:16:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7960 [2026-10-22 03:16:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4209 [2026-10-22 03:16:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7186 [2026-10-22 03:16:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3061 [2026-10-22 03:17:11][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,606 | MFU: 1.8% [2026-10-22 03:17:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0020 [2026-10-22 03:17:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9064 [2026-10-22 03:17:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9998 [2026-10-22 03:17:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9186 [2026-10-22 03:17:16][INFO][training.pretrain_engine] [Ph2|Step 514] loss=4.8995 | ppl=134.22 | aux=0.02507 | lr=2.98e-04 | gnorm=0.676 | tok/s=29,279 | MFU=7.0% | step=143253ms | tokens=1.341B | mem=4.2/32.7GB | cost=$42.15/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 12:52 (129.6h) | Phase3: SKIPPED | SFT: Oct 27 17:37 (134.3h) | DPO (Final): Oct 27 18:11 (134.9h) [2026-10-22 03:17:16][INFO][training.pretrain_engine] Starting global step 515 (accumulating 32 micro-batches)... [2026-10-22 03:17:16][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 03:17:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9533 [2026-10-22 03:17:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8746 [2026-10-22 03:17:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7204 [2026-10-22 03:17:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8202 [2026-10-22 03:17:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9599 [2026-10-22 03:17:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1262 [2026-10-22 03:17:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2159 [2026-10-22 03:17:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6268 [2026-10-22 03:18:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3067 [2026-10-22 03:18:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5067 [2026-10-22 03:18:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2504 [2026-10-22 03:18:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3171 [2026-10-22 03:18:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1377 [2026-10-22 03:18:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1410 [2026-10-22 03:18:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0032 [2026-10-22 03:18:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9027 [2026-10-22 03:18:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9785 [2026-10-22 03:18:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1092 [2026-10-22 03:18:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2859 [2026-10-22 03:18:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7770 [2026-10-22 03:19:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2784 [2026-10-22 03:19:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6151 [2026-10-22 03:19:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7135 [2026-10-22 03:19:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4307 [2026-10-22 03:19:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4949 [2026-10-22 03:19:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8098 [2026-10-22 03:19:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5673 [2026-10-22 03:19:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=3.8699 [2026-10-22 03:19:34][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,615 | MFU: 1.8% [2026-10-22 03:19:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9757 [2026-10-22 03:19:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2650 [2026-10-22 03:19:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4117 [2026-10-22 03:19:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1113 [2026-10-22 03:19:39][INFO][training.pretrain_engine] [Ph2|Step 515] loss=4.9311 | ppl=138.53 | aux=0.02506 | lr=2.98e-04 | gnorm=0.559 | tok/s=29,278 | MFU=7.0% | step=143259ms | tokens=1.342B | mem=4.2/32.7GB | cost=$42.23/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 12:54 (129.6h) | Phase3: SKIPPED | SFT: Oct 27 17:39 (134.3h) | DPO (Final): Oct 27 18:13 (134.9h) [2026-10-22 03:19:39][INFO][training.pretrain_engine] Starting global step 516 (accumulating 32 micro-batches)... [2026-10-22 03:19:39][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 03:19:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0952 [2026-10-22 03:19:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1914 [2026-10-22 03:19:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9115 [2026-10-22 03:19:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0782 [2026-10-22 03:20:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5733 [2026-10-22 03:20:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7408 [2026-10-22 03:20:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9759 [2026-10-22 03:20:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7779 [2026-10-22 03:20:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5533 [2026-10-22 03:20:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4994 [2026-10-22 03:20:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4055 [2026-10-22 03:20:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2693 [2026-10-22 03:20:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.0081 [2026-10-22 03:20:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7934 [2026-10-22 03:20:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6964 [2026-10-22 03:20:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1045 [2026-10-22 03:21:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8334 [2026-10-22 03:21:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3289 [2026-10-22 03:21:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=6.0237 [2026-10-22 03:21:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4688 [2026-10-22 03:21:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1653 [2026-10-22 03:21:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1386 [2026-10-22 03:21:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2813 [2026-10-22 03:21:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5837 [2026-10-22 03:21:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6615 [2026-10-22 03:21:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2990 [2026-10-22 03:21:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6723 [2026-10-22 03:21:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5262 [2026-10-22 03:21:57][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,595 | MFU: 1.8% [2026-10-22 03:22:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4658 [2026-10-22 03:22:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8333 [2026-10-22 03:22:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6892 [2026-10-22 03:22:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2045 [2026-10-22 03:22:02][INFO][training.pretrain_engine] [Ph2|Step 516] loss=4.9271 | ppl=137.98 | aux=0.02507 | lr=2.98e-04 | gnorm=0.637 | tok/s=29,277 | MFU=7.0% | step=143264ms | tokens=1.343B | mem=4.2/32.7GB | cost=$42.32/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 12:56 (129.6h) | Phase3: SKIPPED | SFT: Oct 27 17:41 (134.3h) | DPO (Final): Oct 27 18:15 (134.9h) [2026-10-22 03:22:02][INFO][training.pretrain_engine] Starting global step 517 (accumulating 32 micro-batches)... [2026-10-22 03:22:02][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 03:22:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7434 [2026-10-22 03:22:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5738 [2026-10-22 03:22:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.6821 [2026-10-22 03:22:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5650 [2026-10-22 03:22:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4678 [2026-10-22 03:22:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0845 [2026-10-22 03:22:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0348 [2026-10-22 03:22:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1095 [2026-10-22 03:22:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0278 [2026-10-22 03:22:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3137 [2026-10-22 03:22:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0195 [2026-10-22 03:22:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7620 [2026-10-22 03:23:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0627 [2026-10-22 03:23:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1678 [2026-10-22 03:23:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0749 [2026-10-22 03:23:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2909 [2026-10-22 03:23:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3203 [2026-10-22 03:23:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5033 [2026-10-22 03:23:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1445 [2026-10-22 03:23:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5035 [2026-10-22 03:23:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8340 [2026-10-22 03:23:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0653 [2026-10-22 03:23:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8473 [2026-10-22 03:23:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8453 [2026-10-22 03:24:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0309 [2026-10-22 03:24:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0661 [2026-10-22 03:24:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0228 [2026-10-22 03:24:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3765 [2026-10-22 03:24:21][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,570 | MFU: 1.8% [2026-10-22 03:24:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5365 [2026-10-22 03:24:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7708 [2026-10-22 03:24:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4731 [2026-10-22 03:24:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3446 [2026-10-22 03:24:26][INFO][training.pretrain_engine] [Ph2|Step 517] loss=4.9296 | ppl=138.33 | aux=0.02506 | lr=2.98e-04 | gnorm=0.473 | tok/s=29,275 | MFU=7.0% | step=143273ms | tokens=1.344B | mem=4.2/32.7GB | cost=$42.40/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 12:58 (129.6h) | Phase3: SKIPPED | SFT: Oct 27 17:43 (134.3h) | DPO (Final): Oct 27 18:17 (134.9h) [2026-10-22 03:24:26][INFO][training.pretrain_engine] Starting global step 518 (accumulating 32 micro-batches)... [2026-10-22 03:24:26][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 03:24:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0942 [2026-10-22 03:24:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3056 [2026-10-22 03:24:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7595 [2026-10-22 03:24:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0791 [2026-10-22 03:25:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8936 [2026-10-22 03:25:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1053 [2026-10-22 03:25:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0058 [2026-10-22 03:25:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9703 [2026-10-22 03:25:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8439 [2026-10-22 03:25:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0675 [2026-10-22 03:25:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8937 [2026-10-22 03:25:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2876 [2026-10-22 03:25:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1868 [2026-10-22 03:25:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3045 [2026-10-22 03:25:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5829 [2026-10-22 03:25:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4936 [2026-10-22 03:25:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1192 [2026-10-22 03:25:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3542 [2026-10-22 03:25:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2939 [2026-10-22 03:25:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9132 [2026-10-22 03:26:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8280 [2026-10-22 03:26:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8836 [2026-10-22 03:26:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0849 [2026-10-22 03:26:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8494 [2026-10-22 03:26:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6118 [2026-10-22 03:26:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4251 [2026-10-22 03:26:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3326 [2026-10-22 03:26:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6530 [2026-10-22 03:26:44][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,595 | MFU: 1.8% [2026-10-22 03:26:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8838 [2026-10-22 03:26:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0611 [2026-10-22 03:26:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4085 [2026-10-22 03:26:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0979 [2026-10-22 03:26:50][INFO][training.pretrain_engine] [Ph2|Step 518] loss=4.9161 | ppl=136.46 | aux=0.02506 | lr=2.98e-04 | gnorm=0.590 | tok/s=29,274 | MFU=7.0% | step=143277ms | tokens=1.345B | mem=4.2/32.7GB | cost=$42.48/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 13:00 (129.6h) | Phase3: SKIPPED | SFT: Oct 27 17:45 (134.3h) | DPO (Final): Oct 27 18:19 (134.9h) [2026-10-22 03:26:50][INFO][training.pretrain_engine] Starting global step 519 (accumulating 32 micro-batches)... [2026-10-22 03:26:50][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 03:27:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6045 [2026-10-22 03:27:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1193 [2026-10-22 03:27:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1272 [2026-10-22 03:27:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5042 [2026-10-22 03:27:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9250 [2026-10-22 03:27:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1569 [2026-10-22 03:27:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0635 [2026-10-22 03:27:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5448 [2026-10-22 03:27:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0146 [2026-10-22 03:27:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9258 [2026-10-22 03:27:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8713 [2026-10-22 03:27:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1418 [2026-10-22 03:28:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3390 [2026-10-22 03:28:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8512 [2026-10-22 03:28:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4844 [2026-10-22 03:28:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9173 [2026-10-22 03:28:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8818 [2026-10-22 03:28:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8385 [2026-10-22 03:28:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8218 [2026-10-22 03:28:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7576 [2026-10-22 03:28:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2381 [2026-10-22 03:28:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8089 [2026-10-22 03:28:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7452 [2026-10-22 03:28:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6783 [2026-10-22 03:28:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6928 [2026-10-22 03:28:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.1248 [2026-10-22 03:28:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2245 [2026-10-22 03:28:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1998 [2026-10-22 03:29:08][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,588 | MFU: 1.8% [2026-10-22 03:29:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5516 [2026-10-22 03:29:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6319 [2026-10-22 03:29:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3842 [2026-10-22 03:29:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8400 [2026-10-22 03:29:13][INFO][training.pretrain_engine] [Ph2|Step 519] loss=4.9823 | ppl=145.80 | aux=0.02505 | lr=2.98e-04 | gnorm=0.484 | tok/s=29,273 | MFU=7.0% | step=143284ms | tokens=1.346B | mem=4.2/32.7GB | cost=$42.56/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 13:03 (129.6h) | Phase3: SKIPPED | SFT: Oct 27 17:47 (134.3h) | DPO (Final): Oct 27 18:21 (134.9h) [2026-10-22 03:29:13][INFO][training.pretrain_engine] Starting global step 520 (accumulating 32 micro-batches)... [2026-10-22 03:29:13][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 03:29:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2404 [2026-10-22 03:29:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8519 [2026-10-22 03:29:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9657 [2026-10-22 03:29:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9463 [2026-10-22 03:29:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.0754 [2026-10-22 03:29:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.7721 [2026-10-22 03:29:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3914 [2026-10-22 03:29:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.8413 [2026-10-22 03:30:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1121 [2026-10-22 03:30:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2569 [2026-10-22 03:30:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7007 [2026-10-22 03:30:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8810 [2026-10-22 03:30:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7497 [2026-10-22 03:30:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0188 [2026-10-22 03:30:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2207 [2026-10-22 03:30:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7586 [2026-10-22 03:30:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9562 [2026-10-22 03:30:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8878 [2026-10-22 03:30:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1825 [2026-10-22 03:30:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0728 [2026-10-22 03:31:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8870 [2026-10-22 03:31:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5911 [2026-10-22 03:31:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8331 [2026-10-22 03:31:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.1551 [2026-10-22 03:31:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0718 [2026-10-22 03:31:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8838 [2026-10-22 03:31:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8410 [2026-10-22 03:31:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9163 [2026-10-22 03:31:31][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,584 | MFU: 1.8% [2026-10-22 03:31:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7944 [2026-10-22 03:31:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1616 [2026-10-22 03:31:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0824 [2026-10-22 03:31:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0973 [2026-10-22 03:31:36][INFO][training.pretrain_engine] [Ph2|Step 520] loss=4.7767 | ppl=118.72 | aux=0.02506 | lr=2.98e-04 | gnorm=0.574 | tok/s=29,272 | MFU=7.0% | step=143287ms | tokens=1.347B | mem=4.2/32.7GB | cost=$42.65/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 13:05 (129.6h) | Phase3: SKIPPED | SFT: Oct 27 17:49 (134.3h) | DPO (Final): Oct 27 18:23 (134.9h) [2026-10-22 03:31:36][INFO][training.pretrain_engine] Starting global step 521 (accumulating 32 micro-batches)... [2026-10-22 03:31:36][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 03:31:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1095 [2026-10-22 03:31:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6550 [2026-10-22 03:31:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2570 [2026-10-22 03:31:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8224 [2026-10-22 03:32:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6331 [2026-10-22 03:32:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8036 [2026-10-22 03:32:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8864 [2026-10-22 03:32:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4485 [2026-10-22 03:32:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7585 [2026-10-22 03:32:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7660 [2026-10-22 03:32:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8092 [2026-10-22 03:32:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7323 [2026-10-22 03:32:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9161 [2026-10-22 03:32:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5162 [2026-10-22 03:32:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.0530 [2026-10-22 03:32:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1908 [2026-10-22 03:33:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2002 [2026-10-22 03:33:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1616 [2026-10-22 03:33:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6439 [2026-10-22 03:33:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8356 [2026-10-22 03:33:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5282 [2026-10-22 03:33:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6786 [2026-10-22 03:33:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7149 [2026-10-22 03:33:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9645 [2026-10-22 03:33:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1213 [2026-10-22 03:33:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0501 [2026-10-22 03:33:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1051 [2026-10-22 03:33:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9201 [2026-10-22 03:33:55][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,592 | MFU: 1.8% [2026-10-22 03:34:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9620 [2026-10-22 03:34:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2347 [2026-10-22 03:34:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1327 [2026-10-22 03:34:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2204 [2026-10-22 03:34:00][INFO][training.pretrain_engine] [Ph2|Step 521] loss=4.9018 | ppl=134.53 | aux=0.02506 | lr=2.98e-04 | gnorm=0.516 | tok/s=29,273 | MFU=7.0% | step=143284ms | tokens=1.348B | mem=4.2/32.7GB | cost=$42.73/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 13:06 (129.5h) | Phase3: SKIPPED | SFT: Oct 27 17:51 (134.3h) | DPO (Final): Oct 27 18:25 (134.9h) [2026-10-22 03:34:00][INFO][training.pretrain_engine] Starting global step 522 (accumulating 32 micro-batches)... [2026-10-22 03:34:00][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 03:34:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5062 [2026-10-22 03:34:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9486 [2026-10-22 03:34:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4832 [2026-10-22 03:34:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7646 [2026-10-22 03:34:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7564 [2026-10-22 03:34:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7945 [2026-10-22 03:34:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9363 [2026-10-22 03:34:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9387 [2026-10-22 03:34:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7631 [2026-10-22 03:34:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0014 [2026-10-22 03:34:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0991 [2026-10-22 03:34:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0397 [2026-10-22 03:35:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2332 [2026-10-22 03:35:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1201 [2026-10-22 03:35:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2622 [2026-10-22 03:35:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0098 [2026-10-22 03:35:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3564 [2026-10-22 03:35:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0587 [2026-10-22 03:35:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2359 [2026-10-22 03:35:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3596 [2026-10-22 03:35:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4744 [2026-10-22 03:35:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6106 [2026-10-22 03:35:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3651 [2026-10-22 03:35:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1812 [2026-10-22 03:36:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3791 [2026-10-22 03:36:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3351 [2026-10-22 03:36:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0536 [2026-10-22 03:36:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4235 [2026-10-22 03:36:18][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,598 | MFU: 1.8% [2026-10-22 03:36:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1964 [2026-10-22 03:36:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8544 [2026-10-22 03:36:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1970 [2026-10-22 03:36:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1163 [2026-10-22 03:36:23][INFO][training.pretrain_engine] [Ph2|Step 522] loss=5.0518 | ppl=156.30 | aux=0.02506 | lr=2.98e-04 | gnorm=0.439 | tok/s=29,273 | MFU=7.0% | step=143282ms | tokens=1.350B | mem=4.2/32.7GB | cost=$42.81/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 13:08 (129.5h) | Phase3: SKIPPED | SFT: Oct 27 17:53 (134.3h) | DPO (Final): Oct 27 18:27 (134.8h) [2026-10-22 03:36:26][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step522 | Tokens: 1.350B | Spent: $42.81 | Remaining: $957.19 [2026-10-22 03:36:26][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step522 [2026-10-22 03:36:26][INFO][training.pretrain_engine] Starting global step 523 (accumulating 32 micro-batches)... [2026-10-22 03:36:26][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 03:36:26][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-22 03:36:26][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-22 03:36:26][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-22 03:36:26][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step522 to Phase-Technologies/forge-500m-pretrain (path: phase2_step522)... [2026-10-22 03:36:26][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-22 03:36:36][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-22 03:36:36][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step522 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-22 03:36:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5580 [2026-10-22 03:36:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8398 [2026-10-22 03:36:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1627 [2026-10-22 03:36:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9499 [2026-10-22 03:36:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7035 [2026-10-22 03:36:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7704 [2026-10-22 03:36:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6776 [2026-10-22 03:37:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6399 [2026-10-22 03:37:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9382 [2026-10-22 03:37:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6753 [2026-10-22 03:37:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8763 [2026-10-22 03:37:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0679 [2026-10-22 03:37:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9916 [2026-10-22 03:37:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4801 [2026-10-22 03:37:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1158 [2026-10-22 03:37:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0354 [2026-10-22 03:37:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4360 [2026-10-22 03:37:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4420 [2026-10-22 03:37:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8359 [2026-10-22 03:37:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0032 [2026-10-22 03:38:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1093 [2026-10-22 03:38:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9772 [2026-10-22 03:38:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6086 [2026-10-22 03:38:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0985 [2026-10-22 03:38:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2918 [2026-10-22 03:38:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8086 [2026-10-22 03:38:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3024 [2026-10-22 03:38:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2818 [2026-10-22 03:38:44][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,618 | MFU: 1.8% [2026-10-22 03:38:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9165 [2026-10-22 03:38:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8554 [2026-10-22 03:38:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9627 [2026-10-22 03:38:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2658 [2026-10-22 03:38:48][INFO][training.pretrain_engine] [Ph2|Step 523] loss=4.9120 | ppl=135.92 | aux=0.02506 | lr=2.98e-04 | gnorm=0.475 | tok/s=29,277 | MFU=7.0% | step=143264ms | tokens=1.351B | mem=4.2/32.7GB | cost=$42.89/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 13:09 (129.5h) | Phase3: SKIPPED | SFT: Oct 27 17:53 (134.3h) | DPO (Final): Oct 27 18:27 (134.8h) [2026-10-22 03:38:48][INFO][training.pretrain_engine] Starting global step 524 (accumulating 32 micro-batches)... [2026-10-22 03:38:48][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 03:39:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3573 [2026-10-22 03:39:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0992 [2026-10-22 03:39:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9207 [2026-10-22 03:39:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1948 [2026-10-22 03:39:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7701 [2026-10-22 03:39:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3499 [2026-10-22 03:39:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2282 [2026-10-22 03:39:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6540 [2026-10-22 03:39:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1648 [2026-10-22 03:39:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8313 [2026-10-22 03:39:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9382 [2026-10-22 03:39:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7616 [2026-10-22 03:39:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6895 [2026-10-22 03:39:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9703 [2026-10-22 03:39:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8418 [2026-10-22 03:40:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3660 [2026-10-22 03:40:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1014 [2026-10-22 03:40:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0586 [2026-10-22 03:40:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6324 [2026-10-22 03:40:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1541 [2026-10-22 03:40:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0312 [2026-10-22 03:40:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9607 [2026-10-22 03:40:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0211 [2026-10-22 03:40:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8658 [2026-10-22 03:40:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3110 [2026-10-22 03:40:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9762 [2026-10-22 03:40:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8255 [2026-10-22 03:40:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0405 [2026-10-22 03:41:06][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,628 | MFU: 1.8% [2026-10-22 03:41:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0487 [2026-10-22 03:41:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9073 [2026-10-22 03:41:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6527 [2026-10-22 03:41:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9835 [2026-10-22 03:41:11][INFO][training.pretrain_engine] [Ph2|Step 524] loss=4.9615 | ppl=142.81 | aux=0.02506 | lr=2.98e-04 | gnorm=0.457 | tok/s=29,276 | MFU=7.0% | step=143267ms | tokens=1.352B | mem=4.2/32.7GB | cost=$42.97/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 13:11 (129.5h) | Phase3: SKIPPED | SFT: Oct 27 17:55 (134.2h) | DPO (Final): Oct 27 18:29 (134.8h) [2026-10-22 03:41:11][INFO][training.pretrain_engine] Starting global step 525 (accumulating 32 micro-batches)... [2026-10-22 03:41:11][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 03:41:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9152 [2026-10-22 03:41:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2798 [2026-10-22 03:41:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2798 [2026-10-22 03:41:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9224 [2026-10-22 03:41:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2231 [2026-10-22 03:41:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3694 [2026-10-22 03:41:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3793 [2026-10-22 03:41:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3639 [2026-10-22 03:42:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0257 [2026-10-22 03:42:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9896 [2026-10-22 03:42:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3365 [2026-10-22 03:42:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5726 [2026-10-22 03:42:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8641 [2026-10-22 03:42:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9561 [2026-10-22 03:42:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0038 [2026-10-22 03:42:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3911 [2026-10-22 03:42:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1819 [2026-10-22 03:42:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1653 [2026-10-22 03:42:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2092 [2026-10-22 03:42:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4213 [2026-10-22 03:42:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4917 [2026-10-22 03:42:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6073 [2026-10-22 03:42:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5161 [2026-10-22 03:42:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2306 [2026-10-22 03:43:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2391 [2026-10-22 03:43:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0640 [2026-10-22 03:43:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7688 [2026-10-22 03:43:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2036 [2026-10-22 03:43:29][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,624 | MFU: 1.8% [2026-10-22 03:43:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.0302 [2026-10-22 03:43:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=3.9882 [2026-10-22 03:43:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=3.5476 [2026-10-22 03:43:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2571 [2026-10-22 03:43:34][INFO][training.pretrain_engine] [Ph2|Step 525] loss=5.0429 | ppl=154.93 | aux=0.02505 | lr=2.98e-04 | gnorm=0.480 | tok/s=29,279 | MFU=7.0% | step=143254ms | tokens=1.353B | mem=4.2/32.7GB | cost=$43.06/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 13:12 (129.5h) | Phase3: SKIPPED | SFT: Oct 27 17:56 (134.2h) | DPO (Final): Oct 27 18:30 (134.8h) [2026-10-22 03:43:34][INFO][training.pretrain_engine] Starting global step 526 (accumulating 32 micro-batches)... [2026-10-22 03:43:34][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 03:43:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0791 [2026-10-22 03:43:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8768 [2026-10-22 03:43:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9691 [2026-10-22 03:43:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2459 [2026-10-22 03:44:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1956 [2026-10-22 03:44:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3820 [2026-10-22 03:44:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1432 [2026-10-22 03:44:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1629 [2026-10-22 03:44:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3714 [2026-10-22 03:44:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4911 [2026-10-22 03:44:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6553 [2026-10-22 03:44:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4750 [2026-10-22 03:44:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4373 [2026-10-22 03:44:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8405 [2026-10-22 03:44:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1768 [2026-10-22 03:44:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8098 [2026-10-22 03:45:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0408 [2026-10-22 03:45:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1376 [2026-10-22 03:45:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8759 [2026-10-22 03:45:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2750 [2026-10-22 03:45:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6364 [2026-10-22 03:45:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0416 [2026-10-22 03:45:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0387 [2026-10-22 03:45:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6938 [2026-10-22 03:45:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0195 [2026-10-22 03:45:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8874 [2026-10-22 03:45:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9728 [2026-10-22 03:45:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7737 [2026-10-22 03:45:52][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,605 | MFU: 1.8% [2026-10-22 03:45:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2719 [2026-10-22 03:45:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8174 [2026-10-22 03:45:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6664 [2026-10-22 03:45:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6488 [2026-10-22 03:45:57][INFO][training.pretrain_engine] [Ph2|Step 526] loss=4.9626 | ppl=142.97 | aux=0.02506 | lr=2.98e-04 | gnorm=0.520 | tok/s=29,280 | MFU=7.0% | step=143250ms | tokens=1.354B | mem=4.2/32.7GB | cost=$43.14/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 13:13 (129.5h) | Phase3: SKIPPED | SFT: Oct 27 17:58 (134.2h) | DPO (Final): Oct 27 18:32 (134.8h) [2026-10-22 03:45:57][INFO][training.pretrain_engine] Starting global step 527 (accumulating 32 micro-batches)... [2026-10-22 03:45:57][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 03:46:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9941 [2026-10-22 03:46:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1085 [2026-10-22 03:46:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6214 [2026-10-22 03:46:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6864 [2026-10-22 03:46:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0227 [2026-10-22 03:46:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0970 [2026-10-22 03:46:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0072 [2026-10-22 03:46:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6129 [2026-10-22 03:46:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7661 [2026-10-22 03:46:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1517 [2026-10-22 03:46:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8206 [2026-10-22 03:46:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9790 [2026-10-22 03:47:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6917 [2026-10-22 03:47:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1844 [2026-10-22 03:47:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4180 [2026-10-22 03:47:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4591 [2026-10-22 03:47:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3436 [2026-10-22 03:47:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0205 [2026-10-22 03:47:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0995 [2026-10-22 03:47:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9811 [2026-10-22 03:47:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2517 [2026-10-22 03:47:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9008 [2026-10-22 03:47:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6110 [2026-10-22 03:47:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7845 [2026-10-22 03:48:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6176 [2026-10-22 03:48:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1492 [2026-10-22 03:48:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3485 [2026-10-22 03:48:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8109 [2026-10-22 03:48:14][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,622 | MFU: 1.8% [2026-10-22 03:48:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4219 [2026-10-22 03:48:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4589 [2026-10-22 03:48:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3940 [2026-10-22 03:48:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8912 [2026-10-22 03:48:20][INFO][training.pretrain_engine] [Ph2|Step 527] loss=4.9989 | ppl=148.26 | aux=0.02506 | lr=2.98e-04 | gnorm=0.479 | tok/s=29,280 | MFU=7.0% | step=143247ms | tokens=1.355B | mem=4.2/32.7GB | cost=$43.22/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 13:15 (129.5h) | Phase3: SKIPPED | SFT: Oct 27 17:59 (134.2h) | DPO (Final): Oct 27 18:34 (134.8h) [2026-10-22 03:48:20][INFO][training.pretrain_engine] Starting global step 528 (accumulating 32 micro-batches)... [2026-10-22 03:48:20][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 03:48:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8163 [2026-10-22 03:48:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5392 [2026-10-22 03:48:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9379 [2026-10-22 03:48:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5386 [2026-10-22 03:48:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8777 [2026-10-22 03:48:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0547 [2026-10-22 03:48:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1253 [2026-10-22 03:48:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2777 [2026-10-22 03:49:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0504 [2026-10-22 03:49:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7771 [2026-10-22 03:49:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8453 [2026-10-22 03:49:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2526 [2026-10-22 03:49:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5434 [2026-10-22 03:49:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6640 [2026-10-22 03:49:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7011 [2026-10-22 03:49:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3437 [2026-10-22 03:49:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.2802 [2026-10-22 03:49:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7091 [2026-10-22 03:49:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4609 [2026-10-22 03:49:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5870 [2026-10-22 03:50:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1797 [2026-10-22 03:50:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2412 [2026-10-22 03:50:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7953 [2026-10-22 03:50:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6937 [2026-10-22 03:50:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7564 [2026-10-22 03:50:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5824 [2026-10-22 03:50:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6024 [2026-10-22 03:50:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3862 [2026-10-22 03:50:37][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,617 | MFU: 1.8% [2026-10-22 03:50:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0470 [2026-10-22 03:50:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7933 [2026-10-22 03:50:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9218 [2026-10-22 03:50:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2285 [2026-10-22 03:50:42][INFO][training.pretrain_engine] [Ph2|Step 528] loss=4.9123 | ppl=135.95 | aux=0.02506 | lr=2.98e-04 | gnorm=0.559 | tok/s=29,282 | MFU=7.0% | step=143239ms | tokens=1.356B | mem=4.2/32.7GB | cost=$43.30/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 13:16 (129.4h) | Phase3: SKIPPED | SFT: Oct 27 18:01 (134.2h) | DPO (Final): Oct 27 18:35 (134.7h) [2026-10-22 03:50:42][INFO][training.pretrain_engine] Starting global step 529 (accumulating 32 micro-batches)... [2026-10-22 03:50:42][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 03:51:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4780 [2026-10-22 03:51:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9887 [2026-10-22 03:51:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1599 [2026-10-22 03:51:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0272 [2026-10-22 03:51:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2674 [2026-10-22 03:51:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2072 [2026-10-22 03:51:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1911 [2026-10-22 03:51:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0506 [2026-10-22 03:51:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0721 [2026-10-22 03:51:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6817 [2026-10-22 03:51:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0578 [2026-10-22 03:51:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6121 [2026-10-22 03:51:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0835 [2026-10-22 03:51:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0361 [2026-10-22 03:51:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8954 [2026-10-22 03:51:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1232 [2026-10-22 03:52:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9543 [2026-10-22 03:52:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0979 [2026-10-22 03:52:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1093 [2026-10-22 03:52:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7332 [2026-10-22 03:52:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1943 [2026-10-22 03:52:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9781 [2026-10-22 03:52:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8436 [2026-10-22 03:52:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1440 [2026-10-22 03:52:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3076 [2026-10-22 03:52:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1386 [2026-10-22 03:52:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2424 [2026-10-22 03:52:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6565 [2026-10-22 03:53:00][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,598 | MFU: 1.8% [2026-10-22 03:53:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0236 [2026-10-22 03:53:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9709 [2026-10-22 03:53:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3736 [2026-10-22 03:53:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9119 [2026-10-22 03:53:05][INFO][training.pretrain_engine] [Ph2|Step 529] loss=4.9475 | ppl=140.82 | aux=0.02506 | lr=2.98e-04 | gnorm=0.715 | tok/s=29,282 | MFU=7.0% | step=143236ms | tokens=1.357B | mem=4.2/32.7GB | cost=$43.38/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 13:18 (129.4h) | Phase3: SKIPPED | SFT: Oct 27 18:02 (134.2h) | DPO (Final): Oct 27 18:37 (134.7h) [2026-10-22 03:53:05][INFO][training.pretrain_engine] Starting global step 530 (accumulating 32 micro-batches)... [2026-10-22 03:53:05][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 03:53:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6797 [2026-10-22 03:53:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9802 [2026-10-22 03:53:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1218 [2026-10-22 03:53:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8327 [2026-10-22 03:53:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9437 [2026-10-22 03:53:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0496 [2026-10-22 03:53:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1818 [2026-10-22 03:53:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9614 [2026-10-22 03:53:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5796 [2026-10-22 03:53:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.1912 [2026-10-22 03:53:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6372 [2026-10-22 03:53:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8242 [2026-10-22 03:54:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2484 [2026-10-22 03:54:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0920 [2026-10-22 03:54:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7893 [2026-10-22 03:54:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7886 [2026-10-22 03:54:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4582 [2026-10-22 03:54:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.2566 [2026-10-22 03:54:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5817 [2026-10-22 03:54:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.1974 [2026-10-22 03:54:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4614 [2026-10-22 03:54:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5752 [2026-10-22 03:54:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0099 [2026-10-22 03:54:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2086 [2026-10-22 03:55:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9053 [2026-10-22 03:55:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1603 [2026-10-22 03:55:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2698 [2026-10-22 03:55:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6387 [2026-10-22 03:55:23][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,604 | MFU: 1.8% [2026-10-22 03:55:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6482 [2026-10-22 03:55:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=3.9986 [2026-10-22 03:55:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9531 [2026-10-22 03:55:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5779 [2026-10-22 03:55:28][INFO][training.pretrain_engine] [Ph2|Step 530] loss=4.8997 | ppl=134.24 | aux=0.02506 | lr=2.98e-04 | gnorm=0.844 | tok/s=29,283 | MFU=7.0% | step=143233ms | tokens=1.358B | mem=4.2/32.7GB | cost=$43.46/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 13:19 (129.4h) | Phase3: SKIPPED | SFT: Oct 27 18:04 (134.2h) | DPO (Final): Oct 27 18:38 (134.7h) [2026-10-22 03:55:28][INFO][training.pretrain_engine] Starting global step 531 (accumulating 32 micro-batches)... [2026-10-22 03:55:28][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 03:55:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9139 [2026-10-22 03:55:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0478 [2026-10-22 03:55:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1889 [2026-10-22 03:55:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9656 [2026-10-22 03:56:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6214 [2026-10-22 03:56:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5453 [2026-10-22 03:56:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2891 [2026-10-22 03:56:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.0708 [2026-10-22 03:56:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1717 [2026-10-22 03:56:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2605 [2026-10-22 03:56:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4729 [2026-10-22 03:56:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6891 [2026-10-22 03:56:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8232 [2026-10-22 03:56:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5579 [2026-10-22 03:56:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6319 [2026-10-22 03:56:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8301 [2026-10-22 03:56:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9421 [2026-10-22 03:56:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9862 [2026-10-22 03:56:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8166 [2026-10-22 03:56:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1145 [2026-10-22 03:57:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0690 [2026-10-22 03:57:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4409 [2026-10-22 03:57:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7936 [2026-10-22 03:57:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9978 [2026-10-22 03:57:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0378 [2026-10-22 03:57:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4268 [2026-10-22 03:57:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0906 [2026-10-22 03:57:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2855 [2026-10-22 03:57:46][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,603 | MFU: 1.8% [2026-10-22 03:57:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1437 [2026-10-22 03:57:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6827 [2026-10-22 03:57:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5030 [2026-10-22 03:57:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9905 [2026-10-22 03:57:51][INFO][training.pretrain_engine] [Ph2|Step 531] loss=4.9641 | ppl=143.19 | aux=0.02506 | lr=2.98e-04 | gnorm=0.969 | tok/s=29,283 | MFU=7.0% | step=143231ms | tokens=1.359B | mem=4.2/32.7GB | cost=$43.55/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 13:21 (129.4h) | Phase3: SKIPPED | SFT: Oct 27 18:06 (134.1h) | DPO (Final): Oct 27 18:40 (134.7h) [2026-10-22 03:57:55][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step531 | Tokens: 1.359B | Spent: $43.55 | Remaining: $956.45 [2026-10-22 03:57:55][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step531 [2026-10-22 03:57:55][INFO][training.pretrain_engine] Starting global step 532 (accumulating 32 micro-batches)... [2026-10-22 03:57:55][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 03:57:55][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-22 03:57:55][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-22 03:57:55][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-22 03:57:55][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step531 to Phase-Technologies/forge-500m-pretrain (path: phase2_step531)... [2026-10-22 03:57:55][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-22 03:58:04][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-22 03:58:04][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step531 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-22 03:58:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9580 [2026-10-22 03:58:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0889 [2026-10-22 03:58:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1958 [2026-10-22 03:58:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8586 [2026-10-22 03:58:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5103 [2026-10-22 03:58:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3184 [2026-10-22 03:58:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1979 [2026-10-22 03:58:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6244 [2026-10-22 03:58:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6935 [2026-10-22 03:58:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9458 [2026-10-22 03:58:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5952 [2026-10-22 03:58:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9210 [2026-10-22 03:59:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8838 [2026-10-22 03:59:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8168 [2026-10-22 03:59:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6987 [2026-10-22 03:59:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2429 [2026-10-22 03:59:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1397 [2026-10-22 03:59:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4480 [2026-10-22 03:59:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6056 [2026-10-22 03:59:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8325 [2026-10-22 03:59:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3515 [2026-10-22 03:59:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1588 [2026-10-22 03:59:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1106 [2026-10-22 03:59:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9917 [2026-10-22 03:59:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4222 [2026-10-22 03:59:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1114 [2026-10-22 03:59:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9873 [2026-10-22 03:59:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7880 [2026-10-22 04:00:13][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,588 | MFU: 1.8% [2026-10-22 04:00:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3908 [2026-10-22 04:00:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4805 [2026-10-22 04:00:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=3.8784 [2026-10-22 04:00:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4322 [2026-10-22 04:00:17][INFO][training.pretrain_engine] [Ph2|Step 532] loss=5.0176 | ppl=151.04 | aux=0.02506 | lr=2.98e-04 | gnorm=1.016 | tok/s=29,284 | MFU=7.0% | step=143229ms | tokens=1.360B | mem=4.2/32.7GB | cost=$43.63/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 13:23 (129.4h) | Phase3: SKIPPED | SFT: Oct 27 18:07 (134.1h) | DPO (Final): Oct 27 18:42 (134.7h) [2026-10-22 04:00:17][INFO][training.pretrain_engine] Starting global step 533 (accumulating 32 micro-batches)... [2026-10-22 04:00:17][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 04:00:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8930 [2026-10-22 04:00:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1763 [2026-10-22 04:00:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4761 [2026-10-22 04:00:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2388 [2026-10-22 04:00:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5849 [2026-10-22 04:00:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3642 [2026-10-22 04:00:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9973 [2026-10-22 04:00:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7602 [2026-10-22 04:01:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3349 [2026-10-22 04:01:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6825 [2026-10-22 04:01:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8616 [2026-10-22 04:01:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6148 [2026-10-22 04:01:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4640 [2026-10-22 04:01:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2269 [2026-10-22 04:01:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3793 [2026-10-22 04:01:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7140 [2026-10-22 04:01:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9955 [2026-10-22 04:01:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5630 [2026-10-22 04:01:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8155 [2026-10-22 04:01:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9435 [2026-10-22 04:02:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2373 [2026-10-22 04:02:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1800 [2026-10-22 04:02:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3457 [2026-10-22 04:02:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6786 [2026-10-22 04:02:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0145 [2026-10-22 04:02:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8772 [2026-10-22 04:02:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8134 [2026-10-22 04:02:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5576 [2026-10-22 04:02:35][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,603 | MFU: 1.8% [2026-10-22 04:02:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6369 [2026-10-22 04:02:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6131 [2026-10-22 04:02:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6647 [2026-10-22 04:02:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=6.0836 [2026-10-22 04:02:40][INFO][training.pretrain_engine] [Ph2|Step 533] loss=4.9277 | ppl=138.06 | aux=0.02506 | lr=2.98e-04 | gnorm=0.758 | tok/s=29,283 | MFU=7.0% | step=143235ms | tokens=1.361B | mem=4.2/32.7GB | cost=$43.71/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 13:25 (129.4h) | Phase3: SKIPPED | SFT: Oct 27 18:10 (134.1h) | DPO (Final): Oct 27 18:44 (134.7h) [2026-10-22 04:02:40][INFO][training.pretrain_engine] Starting global step 534 (accumulating 32 micro-batches)... [2026-10-22 04:02:40][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 04:02:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6143 [2026-10-22 04:02:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2420 [2026-10-22 04:02:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7399 [2026-10-22 04:02:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1596 [2026-10-22 04:03:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0828 [2026-10-22 04:03:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1630 [2026-10-22 04:03:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7483 [2026-10-22 04:03:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2815 [2026-10-22 04:03:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3681 [2026-10-22 04:03:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7127 [2026-10-22 04:03:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.1343 [2026-10-22 04:03:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=3.9783 [2026-10-22 04:03:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3024 [2026-10-22 04:03:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9070 [2026-10-22 04:03:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4013 [2026-10-22 04:03:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2790 [2026-10-22 04:04:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1922 [2026-10-22 04:04:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2608 [2026-10-22 04:04:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1962 [2026-10-22 04:04:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9810 [2026-10-22 04:04:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4691 [2026-10-22 04:04:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7627 [2026-10-22 04:04:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7064 [2026-10-22 04:04:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4009 [2026-10-22 04:04:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3939 [2026-10-22 04:04:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5924 [2026-10-22 04:04:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0558 [2026-10-22 04:04:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9179 [2026-10-22 04:04:58][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,602 | MFU: 1.8% [2026-10-22 04:05:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3677 [2026-10-22 04:05:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4767 [2026-10-22 04:05:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4277 [2026-10-22 04:05:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5714 [2026-10-22 04:05:04][INFO][training.pretrain_engine] [Ph2|Step 534] loss=4.9478 | ppl=140.87 | aux=0.02506 | lr=2.98e-04 | gnorm=0.816 | tok/s=29,282 | MFU=7.0% | step=143238ms | tokens=1.362B | mem=4.2/32.7GB | cost=$43.79/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 13:27 (129.4h) | Phase3: SKIPPED | SFT: Oct 27 18:12 (134.1h) | DPO (Final): Oct 27 18:46 (134.7h) [2026-10-22 04:05:04][INFO][training.pretrain_engine] Starting global step 535 (accumulating 32 micro-batches)... [2026-10-22 04:05:04][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 04:05:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9420 [2026-10-22 04:05:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1928 [2026-10-22 04:05:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9253 [2026-10-22 04:05:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7863 [2026-10-22 04:05:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2845 [2026-10-22 04:05:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9868 [2026-10-22 04:05:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9762 [2026-10-22 04:05:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8239 [2026-10-22 04:05:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0621 [2026-10-22 04:05:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2009 [2026-10-22 04:05:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9968 [2026-10-22 04:05:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8309 [2026-10-22 04:06:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9731 [2026-10-22 04:06:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0522 [2026-10-22 04:06:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5659 [2026-10-22 04:06:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6243 [2026-10-22 04:06:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2295 [2026-10-22 04:06:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8227 [2026-10-22 04:06:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9655 [2026-10-22 04:06:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2573 [2026-10-22 04:06:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6240 [2026-10-22 04:06:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8931 [2026-10-22 04:06:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8273 [2026-10-22 04:06:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8828 [2026-10-22 04:07:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0546 [2026-10-22 04:07:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3689 [2026-10-22 04:07:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0529 [2026-10-22 04:07:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0083 [2026-10-22 04:07:22][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,606 | MFU: 1.8% [2026-10-22 04:07:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4813 [2026-10-22 04:07:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8973 [2026-10-22 04:07:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9335 [2026-10-22 04:07:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9757 [2026-10-22 04:07:27][INFO][training.pretrain_engine] [Ph2|Step 535] loss=4.8846 | ppl=132.23 | aux=0.02506 | lr=2.98e-04 | gnorm=0.918 | tok/s=29,282 | MFU=7.0% | step=143241ms | tokens=1.363B | mem=4.2/32.7GB | cost=$43.88/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 13:29 (129.4h) | Phase3: SKIPPED | SFT: Oct 27 18:13 (134.1h) | DPO (Final): Oct 27 18:48 (134.7h) [2026-10-22 04:07:27][INFO][training.pretrain_engine] Starting global step 536 (accumulating 32 micro-batches)... [2026-10-22 04:07:27][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 04:07:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5739 [2026-10-22 04:07:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6472 [2026-10-22 04:07:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8280 [2026-10-22 04:07:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7202 [2026-10-22 04:08:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8257 [2026-10-22 04:08:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3614 [2026-10-22 04:08:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7060 [2026-10-22 04:08:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8892 [2026-10-22 04:08:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2853 [2026-10-22 04:08:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3153 [2026-10-22 04:08:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.1447 [2026-10-22 04:08:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4721 [2026-10-22 04:08:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1974 [2026-10-22 04:08:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9596 [2026-10-22 04:08:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1691 [2026-10-22 04:08:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7008 [2026-10-22 04:08:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8009 [2026-10-22 04:08:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9417 [2026-10-22 04:08:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=3.7512 [2026-10-22 04:08:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9759 [2026-10-22 04:09:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1453 [2026-10-22 04:09:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0925 [2026-10-22 04:09:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4529 [2026-10-22 04:09:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9909 [2026-10-22 04:09:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7786 [2026-10-22 04:09:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8927 [2026-10-22 04:09:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9838 [2026-10-22 04:09:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2326 [2026-10-22 04:09:45][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,599 | MFU: 1.8% [2026-10-22 04:09:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6362 [2026-10-22 04:09:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1437 [2026-10-22 04:09:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6800 [2026-10-22 04:09:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6547 [2026-10-22 04:09:50][INFO][training.pretrain_engine] [Ph2|Step 536] loss=5.0219 | ppl=151.70 | aux=0.02507 | lr=2.98e-04 | gnorm=0.785 | tok/s=29,282 | MFU=7.0% | step=143239ms | tokens=1.364B | mem=4.2/32.7GB | cost=$43.96/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 13:31 (129.4h) | Phase3: SKIPPED | SFT: Oct 27 18:15 (134.1h) | DPO (Final): Oct 27 18:49 (134.7h) [2026-10-22 04:09:50][INFO][training.pretrain_engine] Starting global step 537 (accumulating 32 micro-batches)... [2026-10-22 04:09:50][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 04:10:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9757 [2026-10-22 04:10:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2856 [2026-10-22 04:10:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7164 [2026-10-22 04:10:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6284 [2026-10-22 04:10:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4447 [2026-10-22 04:10:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6692 [2026-10-22 04:10:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5974 [2026-10-22 04:10:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4406 [2026-10-22 04:10:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0798 [2026-10-22 04:10:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7141 [2026-10-22 04:10:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6124 [2026-10-22 04:10:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.0563 [2026-10-22 04:11:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5880 [2026-10-22 04:11:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2598 [2026-10-22 04:11:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4985 [2026-10-22 04:11:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9697 [2026-10-22 04:11:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7174 [2026-10-22 04:11:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.2595 [2026-10-22 04:11:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5725 [2026-10-22 04:11:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.1502 [2026-10-22 04:11:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6435 [2026-10-22 04:11:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5190 [2026-10-22 04:11:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0693 [2026-10-22 04:11:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3346 [2026-10-22 04:11:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9674 [2026-10-22 04:11:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6687 [2026-10-22 04:11:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4288 [2026-10-22 04:11:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9768 [2026-10-22 04:12:08][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,603 | MFU: 1.8% [2026-10-22 04:12:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7766 [2026-10-22 04:12:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8411 [2026-10-22 04:12:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0651 [2026-10-22 04:12:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7705 [2026-10-22 04:12:13][INFO][training.pretrain_engine] [Ph2|Step 537] loss=4.8694 | ppl=130.24 | aux=0.02507 | lr=2.98e-04 | gnorm=0.727 | tok/s=29,284 | MFU=7.0% | step=143231ms | tokens=1.365B | mem=4.2/32.7GB | cost=$44.04/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 13:32 (129.3h) | Phase3: SKIPPED | SFT: Oct 27 18:16 (134.1h) | DPO (Final): Oct 27 18:51 (134.6h) [2026-10-22 04:12:13][INFO][training.pretrain_engine] Starting global step 538 (accumulating 32 micro-batches)... [2026-10-22 04:12:13][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 04:12:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0028 [2026-10-22 04:12:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8069 [2026-10-22 04:12:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2236 [2026-10-22 04:12:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9028 [2026-10-22 04:12:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5976 [2026-10-22 04:12:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6026 [2026-10-22 04:12:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4411 [2026-10-22 04:12:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9099 [2026-10-22 04:13:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1865 [2026-10-22 04:13:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9470 [2026-10-22 04:13:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.0570 [2026-10-22 04:13:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2598 [2026-10-22 04:13:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2260 [2026-10-22 04:13:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0672 [2026-10-22 04:13:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8274 [2026-10-22 04:13:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1495 [2026-10-22 04:13:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7940 [2026-10-22 04:13:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3888 [2026-10-22 04:13:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.1906 [2026-10-22 04:13:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8386 [2026-10-22 04:13:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2718 [2026-10-22 04:14:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6589 [2026-10-22 04:14:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8519 [2026-10-22 04:14:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.0990 [2026-10-22 04:14:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7545 [2026-10-22 04:14:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2924 [2026-10-22 04:14:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6561 [2026-10-22 04:14:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0877 [2026-10-22 04:14:31][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,600 | MFU: 1.8% [2026-10-22 04:14:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3786 [2026-10-22 04:14:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7512 [2026-10-22 04:14:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8801 [2026-10-22 04:14:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4693 [2026-10-22 04:14:36][INFO][training.pretrain_engine] [Ph2|Step 538] loss=4.8887 | ppl=132.78 | aux=0.02506 | lr=2.98e-04 | gnorm=0.719 | tok/s=29,284 | MFU=7.0% | step=143229ms | tokens=1.366B | mem=4.2/32.7GB | cost=$44.12/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 13:34 (129.3h) | Phase3: SKIPPED | SFT: Oct 27 18:18 (134.1h) | DPO (Final): Oct 27 18:52 (134.6h) [2026-10-22 04:14:36][INFO][training.pretrain_engine] Starting global step 539 (accumulating 32 micro-batches)... [2026-10-22 04:14:36][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 04:14:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9032 [2026-10-22 04:14:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2037 [2026-10-22 04:14:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6682 [2026-10-22 04:14:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6044 [2026-10-22 04:15:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1137 [2026-10-22 04:15:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0070 [2026-10-22 04:15:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8814 [2026-10-22 04:15:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1410 [2026-10-22 04:15:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3083 [2026-10-22 04:15:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2519 [2026-10-22 04:15:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8652 [2026-10-22 04:15:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0485 [2026-10-22 04:15:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6307 [2026-10-22 04:15:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4318 [2026-10-22 04:15:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1761 [2026-10-22 04:15:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4470 [2026-10-22 04:16:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1892 [2026-10-22 04:16:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6246 [2026-10-22 04:16:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7419 [2026-10-22 04:16:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7673 [2026-10-22 04:16:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3543 [2026-10-22 04:16:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2949 [2026-10-22 04:16:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.1827 [2026-10-22 04:16:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.1322 [2026-10-22 04:16:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6846 [2026-10-22 04:16:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8298 [2026-10-22 04:16:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0028 [2026-10-22 04:16:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1417 [2026-10-22 04:16:54][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,598 | MFU: 1.8% [2026-10-22 04:17:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2320 [2026-10-22 04:17:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9284 [2026-10-22 04:17:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0084 [2026-10-22 04:17:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2572 [2026-10-22 04:17:00][INFO][training.pretrain_engine] [Ph2|Step 539] loss=4.8453 | ppl=127.14 | aux=0.02507 | lr=2.98e-04 | gnorm=0.621 | tok/s=29,284 | MFU=7.0% | step=143228ms | tokens=1.367B | mem=4.2/32.7GB | cost=$44.20/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 13:35 (129.3h) | Phase3: SKIPPED | SFT: Oct 27 18:20 (134.1h) | DPO (Final): Oct 27 18:54 (134.6h) [2026-10-22 04:17:00][INFO][training.pretrain_engine] Starting global step 540 (accumulating 32 micro-batches)... [2026-10-22 04:17:00][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 04:17:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7502 [2026-10-22 04:17:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6389 [2026-10-22 04:17:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.9949 [2026-10-22 04:17:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6760 [2026-10-22 04:17:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7491 [2026-10-22 04:17:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6811 [2026-10-22 04:17:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8517 [2026-10-22 04:17:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4883 [2026-10-22 04:17:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9733 [2026-10-22 04:17:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6245 [2026-10-22 04:17:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8364 [2026-10-22 04:17:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1961 [2026-10-22 04:18:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5916 [2026-10-22 04:18:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8180 [2026-10-22 04:18:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6121 [2026-10-22 04:18:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0096 [2026-10-22 04:18:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0554 [2026-10-22 04:18:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8552 [2026-10-22 04:18:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4117 [2026-10-22 04:18:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7778 [2026-10-22 04:18:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6068 [2026-10-22 04:18:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0244 [2026-10-22 04:18:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7534 [2026-10-22 04:18:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0477 [2026-10-22 04:19:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0110 [2026-10-22 04:19:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1792 [2026-10-22 04:19:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5348 [2026-10-22 04:19:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5554 [2026-10-22 04:19:17][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,613 | MFU: 1.8% [2026-10-22 04:19:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7831 [2026-10-22 04:19:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0881 [2026-10-22 04:19:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1031 [2026-10-22 04:19:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7394 [2026-10-22 04:19:23][INFO][training.pretrain_engine] [Ph2|Step 540] loss=4.8655 | ppl=129.73 | aux=0.02507 | lr=2.98e-04 | gnorm=0.543 | tok/s=29,285 | MFU=7.0% | step=143223ms | tokens=1.368B | mem=4.2/32.7GB | cost=$44.29/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 13:37 (129.3h) | Phase3: SKIPPED | SFT: Oct 27 18:21 (134.0h) | DPO (Final): Oct 27 18:56 (134.6h) [2026-10-22 04:19:26][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step540 | Tokens: 1.368B | Spent: $44.29 | Remaining: $955.71 [2026-10-22 04:19:26][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step540 [2026-10-22 04:19:26][INFO][training.pretrain_engine] Starting global step 541 (accumulating 32 micro-batches)... [2026-10-22 04:19:26][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 04:19:26][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-22 04:19:26][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-22 04:19:26][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-22 04:19:26][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step540 to Phase-Technologies/forge-500m-pretrain (path: phase2_step540)... [2026-10-22 04:19:26][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-22 04:19:35][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-22 04:19:35][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step540 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-22 04:19:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0371 [2026-10-22 04:19:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9798 [2026-10-22 04:19:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8826 [2026-10-22 04:19:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1979 [2026-10-22 04:19:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.9626 [2026-10-22 04:19:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9163 [2026-10-22 04:19:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6915 [2026-10-22 04:20:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6847 [2026-10-22 04:20:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9137 [2026-10-22 04:20:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4744 [2026-10-22 04:20:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0204 [2026-10-22 04:20:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1761 [2026-10-22 04:20:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4795 [2026-10-22 04:20:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8346 [2026-10-22 04:20:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8505 [2026-10-22 04:20:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2268 [2026-10-22 04:20:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7350 [2026-10-22 04:20:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8298 [2026-10-22 04:20:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9958 [2026-10-22 04:20:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0379 [2026-10-22 04:21:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0668 [2026-10-22 04:21:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7919 [2026-10-22 04:21:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4541 [2026-10-22 04:21:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7044 [2026-10-22 04:21:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9121 [2026-10-22 04:21:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0370 [2026-10-22 04:21:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1116 [2026-10-22 04:21:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9640 [2026-10-22 04:21:44][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,596 | MFU: 1.8% [2026-10-22 04:21:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7670 [2026-10-22 04:21:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4351 [2026-10-22 04:21:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1958 [2026-10-22 04:21:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8128 [2026-10-22 04:21:49][INFO][training.pretrain_engine] [Ph2|Step 541] loss=4.9459 | ppl=140.59 | aux=0.02506 | lr=2.98e-04 | gnorm=0.559 | tok/s=29,286 | MFU=7.0% | step=143220ms | tokens=1.369B | mem=4.2/32.7GB | cost=$44.37/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 13:39 (129.3h) | Phase3: SKIPPED | SFT: Oct 27 18:23 (134.0h) | DPO (Final): Oct 27 18:57 (134.6h) [2026-10-22 04:21:49][INFO][training.pretrain_engine] Starting global step 542 (accumulating 32 micro-batches)... [2026-10-22 04:21:49][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 04:22:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6316 [2026-10-22 04:22:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2704 [2026-10-22 04:22:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5024 [2026-10-22 04:22:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2180 [2026-10-22 04:22:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0753 [2026-10-22 04:22:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8186 [2026-10-22 04:22:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1348 [2026-10-22 04:22:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6822 [2026-10-22 04:22:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.1336 [2026-10-22 04:22:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7037 [2026-10-22 04:22:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7749 [2026-10-22 04:22:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7094 [2026-10-22 04:23:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8376 [2026-10-22 04:23:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9541 [2026-10-22 04:23:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6867 [2026-10-22 04:23:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0359 [2026-10-22 04:23:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9613 [2026-10-22 04:23:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2667 [2026-10-22 04:23:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4747 [2026-10-22 04:23:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4085 [2026-10-22 04:23:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8394 [2026-10-22 04:23:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0618 [2026-10-22 04:23:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6738 [2026-10-22 04:23:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0095 [2026-10-22 04:23:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7591 [2026-10-22 04:23:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6126 [2026-10-22 04:23:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8073 [2026-10-22 04:23:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4482 [2026-10-22 04:24:07][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,597 | MFU: 1.8% [2026-10-22 04:24:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6582 [2026-10-22 04:24:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4047 [2026-10-22 04:24:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2338 [2026-10-22 04:24:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8411 [2026-10-22 04:24:12][INFO][training.pretrain_engine] [Ph2|Step 542] loss=4.7854 | ppl=119.75 | aux=0.02505 | lr=2.98e-04 | gnorm=0.516 | tok/s=29,284 | MFU=7.0% | step=143230ms | tokens=1.370B | mem=4.2/32.7GB | cost=$44.45/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 13:41 (129.3h) | Phase3: SKIPPED | SFT: Oct 27 18:25 (134.0h) | DPO (Final): Oct 27 19:00 (134.6h) [2026-10-22 04:24:12][INFO][training.pretrain_engine] Starting global step 543 (accumulating 32 micro-batches)... [2026-10-22 04:24:12][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 04:24:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4324 [2026-10-22 04:24:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1408 [2026-10-22 04:24:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5002 [2026-10-22 04:24:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2235 [2026-10-22 04:24:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1133 [2026-10-22 04:24:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3579 [2026-10-22 04:24:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6859 [2026-10-22 04:24:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7900 [2026-10-22 04:25:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3955 [2026-10-22 04:25:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6870 [2026-10-22 04:25:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5308 [2026-10-22 04:25:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1151 [2026-10-22 04:25:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0272 [2026-10-22 04:25:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.0484 [2026-10-22 04:25:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8440 [2026-10-22 04:25:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0124 [2026-10-22 04:25:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5577 [2026-10-22 04:25:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0425 [2026-10-22 04:25:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3029 [2026-10-22 04:25:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0896 [2026-10-22 04:25:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2193 [2026-10-22 04:25:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2929 [2026-10-22 04:25:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9349 [2026-10-22 04:25:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4343 [2026-10-22 04:26:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.1594 [2026-10-22 04:26:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=3.5313 [2026-10-22 04:26:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2221 [2026-10-22 04:26:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5159 [2026-10-22 04:26:30][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,600 | MFU: 1.8% [2026-10-22 04:26:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.1489 [2026-10-22 04:26:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4704 [2026-10-22 04:26:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3492 [2026-10-22 04:26:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3762 [2026-10-22 04:26:35][INFO][training.pretrain_engine] [Ph2|Step 543] loss=4.7768 | ppl=118.73 | aux=0.02505 | lr=2.98e-04 | gnorm=0.504 | tok/s=29,285 | MFU=7.0% | step=143222ms | tokens=1.372B | mem=4.2/32.7GB | cost=$44.53/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 13:42 (129.3h) | Phase3: SKIPPED | SFT: Oct 27 18:27 (134.0h) | DPO (Final): Oct 27 19:01 (134.6h) [2026-10-22 04:26:35][INFO][training.pretrain_engine] Starting global step 544 (accumulating 32 micro-batches)... [2026-10-22 04:26:35][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 04:26:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0707 [2026-10-22 04:26:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0048 [2026-10-22 04:26:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2571 [2026-10-22 04:26:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0336 [2026-10-22 04:27:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4328 [2026-10-22 04:27:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4474 [2026-10-22 04:27:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9761 [2026-10-22 04:27:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1355 [2026-10-22 04:27:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8858 [2026-10-22 04:27:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6848 [2026-10-22 04:27:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9339 [2026-10-22 04:27:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9656 [2026-10-22 04:27:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4393 [2026-10-22 04:27:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9751 [2026-10-22 04:27:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8192 [2026-10-22 04:27:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0017 [2026-10-22 04:28:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0720 [2026-10-22 04:28:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1511 [2026-10-22 04:28:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8913 [2026-10-22 04:28:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0213 [2026-10-22 04:28:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8255 [2026-10-22 04:28:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6661 [2026-10-22 04:28:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=3.7712 [2026-10-22 04:28:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.0435 [2026-10-22 04:28:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8628 [2026-10-22 04:28:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9990 [2026-10-22 04:28:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0264 [2026-10-22 04:28:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8397 [2026-10-22 04:28:53][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,598 | MFU: 1.8% [2026-10-22 04:28:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1093 [2026-10-22 04:28:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1436 [2026-10-22 04:28:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4019 [2026-10-22 04:28:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3508 [2026-10-22 04:28:57][INFO][training.pretrain_engine] [Ph2|Step 544] loss=5.0057 | ppl=149.25 | aux=0.02505 | lr=2.98e-04 | gnorm=0.516 | tok/s=29,286 | MFU=7.0% | step=143218ms | tokens=1.373B | mem=4.2/32.7GB | cost=$44.61/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 13:44 (129.3h) | Phase3: SKIPPED | SFT: Oct 27 18:28 (134.0h) | DPO (Final): Oct 27 19:02 (134.6h) [2026-10-22 04:28:57][INFO][training.pretrain_engine] Starting global step 545 (accumulating 32 micro-batches)... [2026-10-22 04:28:57][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 04:29:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5747 [2026-10-22 04:29:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9217 [2026-10-22 04:29:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0351 [2026-10-22 04:29:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6919 [2026-10-22 04:29:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6748 [2026-10-22 04:29:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9242 [2026-10-22 04:29:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1545 [2026-10-22 04:29:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2381 [2026-10-22 04:29:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7733 [2026-10-22 04:29:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3853 [2026-10-22 04:29:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2361 [2026-10-22 04:29:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9274 [2026-10-22 04:30:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7578 [2026-10-22 04:30:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0840 [2026-10-22 04:30:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0725 [2026-10-22 04:30:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9838 [2026-10-22 04:30:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1903 [2026-10-22 04:30:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8459 [2026-10-22 04:30:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9421 [2026-10-22 04:30:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1753 [2026-10-22 04:30:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6733 [2026-10-22 04:30:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8801 [2026-10-22 04:30:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9584 [2026-10-22 04:30:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6335 [2026-10-22 04:31:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6227 [2026-10-22 04:31:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0222 [2026-10-22 04:31:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5411 [2026-10-22 04:31:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8073 [2026-10-22 04:31:15][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,603 | MFU: 1.8% [2026-10-22 04:31:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3799 [2026-10-22 04:31:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9221 [2026-10-22 04:31:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8208 [2026-10-22 04:31:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8526 [2026-10-22 04:31:21][INFO][training.pretrain_engine] [Ph2|Step 545] loss=4.9451 | ppl=140.48 | aux=0.02506 | lr=2.98e-04 | gnorm=0.492 | tok/s=29,287 | MFU=7.0% | step=143215ms | tokens=1.374B | mem=4.2/32.7GB | cost=$44.70/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 13:45 (129.2h) | Phase3: SKIPPED | SFT: Oct 27 18:30 (134.0h) | DPO (Final): Oct 27 19:04 (134.6h) [2026-10-22 04:31:21][INFO][training.pretrain_engine] Starting global step 546 (accumulating 32 micro-batches)... [2026-10-22 04:31:21][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 04:31:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0888 [2026-10-22 04:31:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2366 [2026-10-22 04:31:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4768 [2026-10-22 04:31:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0965 [2026-10-22 04:31:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8889 [2026-10-22 04:31:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3658 [2026-10-22 04:31:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7579 [2026-10-22 04:31:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0101 [2026-10-22 04:32:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7841 [2026-10-22 04:32:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9046 [2026-10-22 04:32:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7077 [2026-10-22 04:32:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7257 [2026-10-22 04:32:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1187 [2026-10-22 04:32:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1290 [2026-10-22 04:32:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9884 [2026-10-22 04:32:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7420 [2026-10-22 04:32:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0603 [2026-10-22 04:32:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9967 [2026-10-22 04:32:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6528 [2026-10-22 04:32:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1384 [2026-10-22 04:33:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0989 [2026-10-22 04:33:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3390 [2026-10-22 04:33:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3185 [2026-10-22 04:33:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6200 [2026-10-22 04:33:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5930 [2026-10-22 04:33:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1384 [2026-10-22 04:33:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2267 [2026-10-22 04:33:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9035 [2026-10-22 04:33:38][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,616 | MFU: 1.8% [2026-10-22 04:33:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4829 [2026-10-22 04:33:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5012 [2026-10-22 04:33:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7125 [2026-10-22 04:33:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7438 [2026-10-22 04:33:44][INFO][training.pretrain_engine] [Ph2|Step 546] loss=4.9034 | ppl=134.75 | aux=0.02506 | lr=2.98e-04 | gnorm=0.479 | tok/s=29,288 | MFU=7.0% | step=143210ms | tokens=1.375B | mem=4.2/32.7GB | cost=$44.78/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 13:47 (129.2h) | Phase3: SKIPPED | SFT: Oct 27 18:31 (134.0h) | DPO (Final): Oct 27 19:06 (134.5h) [2026-10-22 04:33:44][INFO][training.pretrain_engine] Starting global step 547 (accumulating 32 micro-batches)... [2026-10-22 04:33:44][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 04:34:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8683 [2026-10-22 04:34:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0723 [2026-10-22 04:34:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8172 [2026-10-22 04:34:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1253 [2026-10-22 04:34:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6852 [2026-10-22 04:34:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7721 [2026-10-22 04:34:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4659 [2026-10-22 04:34:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2555 [2026-10-22 04:34:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8564 [2026-10-22 04:34:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6213 [2026-10-22 04:34:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3721 [2026-10-22 04:34:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0110 [2026-10-22 04:34:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2061 [2026-10-22 04:34:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2016 [2026-10-22 04:34:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9703 [2026-10-22 04:34:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2277 [2026-10-22 04:35:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9051 [2026-10-22 04:35:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0369 [2026-10-22 04:35:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2752 [2026-10-22 04:35:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7144 [2026-10-22 04:35:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8758 [2026-10-22 04:35:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9859 [2026-10-22 04:35:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2944 [2026-10-22 04:35:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0137 [2026-10-22 04:35:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1429 [2026-10-22 04:35:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4409 [2026-10-22 04:35:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0765 [2026-10-22 04:35:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4334 [2026-10-22 04:36:02][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,600 | MFU: 1.8% [2026-10-22 04:36:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8484 [2026-10-22 04:36:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8424 [2026-10-22 04:36:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9245 [2026-10-22 04:36:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0120 [2026-10-22 04:36:07][INFO][training.pretrain_engine] [Ph2|Step 547] loss=5.0179 | ppl=151.09 | aux=0.02506 | lr=2.98e-04 | gnorm=0.471 | tok/s=29,288 | MFU=7.0% | step=143207ms | tokens=1.376B | mem=4.2/32.7GB | cost=$44.86/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 13:49 (129.2h) | Phase3: SKIPPED | SFT: Oct 27 18:33 (134.0h) | DPO (Final): Oct 27 19:07 (134.5h) [2026-10-22 04:36:07][INFO][training.pretrain_engine] Starting global step 548 (accumulating 32 micro-batches)... [2026-10-22 04:36:07][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 04:36:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5489 [2026-10-22 04:36:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2871 [2026-10-22 04:36:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6011 [2026-10-22 04:36:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4033 [2026-10-22 04:36:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6642 [2026-10-22 04:36:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8187 [2026-10-22 04:36:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8703 [2026-10-22 04:36:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4361 [2026-10-22 04:37:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2385 [2026-10-22 04:37:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5267 [2026-10-22 04:37:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5030 [2026-10-22 04:37:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2591 [2026-10-22 04:37:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8831 [2026-10-22 04:37:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0319 [2026-10-22 04:37:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7056 [2026-10-22 04:37:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8432 [2026-10-22 04:37:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6943 [2026-10-22 04:37:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9183 [2026-10-22 04:37:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.2235 [2026-10-22 04:37:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6861 [2026-10-22 04:37:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3299 [2026-10-22 04:37:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0099 [2026-10-22 04:37:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5447 [2026-10-22 04:37:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4938 [2026-10-22 04:38:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0645 [2026-10-22 04:38:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7833 [2026-10-22 04:38:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9640 [2026-10-22 04:38:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0882 [2026-10-22 04:38:25][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,599 | MFU: 1.8% [2026-10-22 04:38:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8155 [2026-10-22 04:38:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3099 [2026-10-22 04:38:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7793 [2026-10-22 04:38:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5446 [2026-10-22 04:38:30][INFO][training.pretrain_engine] [Ph2|Step 548] loss=4.7967 | ppl=121.11 | aux=0.02507 | lr=2.98e-04 | gnorm=0.523 | tok/s=29,290 | MFU=7.0% | step=143199ms | tokens=1.377B | mem=4.2/32.7GB | cost=$44.94/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 13:50 (129.2h) | Phase3: SKIPPED | SFT: Oct 27 18:34 (133.9h) | DPO (Final): Oct 27 19:09 (134.5h) [2026-10-22 04:38:30][INFO][training.pretrain_engine] Starting global step 549 (accumulating 32 micro-batches)... [2026-10-22 04:38:30][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 04:38:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6446 [2026-10-22 04:38:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.1524 [2026-10-22 04:38:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.0794 [2026-10-22 04:38:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8661 [2026-10-22 04:39:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9517 [2026-10-22 04:39:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9524 [2026-10-22 04:39:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0565 [2026-10-22 04:39:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1040 [2026-10-22 04:39:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7248 [2026-10-22 04:39:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1908 [2026-10-22 04:39:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6545 [2026-10-22 04:39:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5162 [2026-10-22 04:39:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.1179 [2026-10-22 04:39:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4948 [2026-10-22 04:39:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3111 [2026-10-22 04:39:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4555 [2026-10-22 04:39:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3567 [2026-10-22 04:40:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1189 [2026-10-22 04:40:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1554 [2026-10-22 04:40:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0214 [2026-10-22 04:40:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6014 [2026-10-22 04:40:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6338 [2026-10-22 04:40:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2415 [2026-10-22 04:40:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4201 [2026-10-22 04:40:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8322 [2026-10-22 04:40:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8855 [2026-10-22 04:40:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0549 [2026-10-22 04:40:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2102 [2026-10-22 04:40:48][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,581 | MFU: 1.8% [2026-10-22 04:40:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5949 [2026-10-22 04:40:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2176 [2026-10-22 04:40:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8026 [2026-10-22 04:40:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8034 [2026-10-22 04:40:53][INFO][training.pretrain_engine] [Ph2|Step 549] loss=4.8696 | ppl=130.27 | aux=0.02506 | lr=2.98e-04 | gnorm=0.449 | tok/s=29,292 | MFU=7.0% | step=143191ms | tokens=1.378B | mem=4.2/32.7GB | cost=$45.02/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 13:51 (129.2h) | Phase3: SKIPPED | SFT: Oct 27 18:36 (133.9h) | DPO (Final): Oct 27 19:10 (134.5h) [2026-10-22 04:40:57][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step549 | Tokens: 1.378B | Spent: $45.03 | Remaining: $954.97 [2026-10-22 04:40:57][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step549 [2026-10-22 04:40:57][INFO][training.pretrain_engine] Starting global step 550 (accumulating 32 micro-batches)... [2026-10-22 04:40:57][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 04:40:57][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-22 04:40:57][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-22 04:40:57][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-22 04:40:57][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step549 to Phase-Technologies/forge-500m-pretrain (path: phase2_step549)... [2026-10-22 04:40:57][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-22 04:41:06][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-22 04:41:06][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step549 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-22 04:41:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4905 [2026-10-22 04:41:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.0971 [2026-10-22 04:41:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.1772 [2026-10-22 04:41:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8185 [2026-10-22 04:41:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5028 [2026-10-22 04:41:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1705 [2026-10-22 04:41:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2903 [2026-10-22 04:41:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0621 [2026-10-22 04:41:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3281 [2026-10-22 04:41:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6108 [2026-10-22 04:41:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.1652 [2026-10-22 04:41:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6783 [2026-10-22 04:42:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5070 [2026-10-22 04:42:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9209 [2026-10-22 04:42:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0360 [2026-10-22 04:42:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7475 [2026-10-22 04:42:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9633 [2026-10-22 04:42:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8109 [2026-10-22 04:42:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5689 [2026-10-22 04:42:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6613 [2026-10-22 04:42:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9265 [2026-10-22 04:42:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6877 [2026-10-22 04:42:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8763 [2026-10-22 04:42:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6940 [2026-10-22 04:42:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8465 [2026-10-22 04:42:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1229 [2026-10-22 04:42:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0737 [2026-10-22 04:43:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3031 [2026-10-22 04:43:15][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,590 | MFU: 1.8% [2026-10-22 04:43:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2252 [2026-10-22 04:43:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8546 [2026-10-22 04:43:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6429 [2026-10-22 04:43:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9955 [2026-10-22 04:43:19][INFO][training.pretrain_engine] [Ph2|Step 550] loss=4.9060 | ppl=135.10 | aux=0.02505 | lr=2.98e-04 | gnorm=0.475 | tok/s=29,294 | MFU=7.0% | step=143178ms | tokens=1.379B | mem=4.2/32.7GB | cost=$45.11/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 13:52 (129.2h) | Phase3: SKIPPED | SFT: Oct 27 18:37 (133.9h) | DPO (Final): Oct 27 19:11 (134.5h) [2026-10-22 04:43:19][INFO][training.pretrain_engine] Starting global step 551 (accumulating 32 micro-batches)... [2026-10-22 04:43:19][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 04:43:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3178 [2026-10-22 04:43:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8965 [2026-10-22 04:43:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9846 [2026-10-22 04:43:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2126 [2026-10-22 04:43:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5179 [2026-10-22 04:43:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7669 [2026-10-22 04:43:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1674 [2026-10-22 04:43:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4169 [2026-10-22 04:44:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9422 [2026-10-22 04:44:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2921 [2026-10-22 04:44:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2475 [2026-10-22 04:44:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7715 [2026-10-22 04:44:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3556 [2026-10-22 04:44:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1558 [2026-10-22 04:44:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1847 [2026-10-22 04:44:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2635 [2026-10-22 04:44:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6874 [2026-10-22 04:44:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5767 [2026-10-22 04:44:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3400 [2026-10-22 04:44:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.1967 [2026-10-22 04:45:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6090 [2026-10-22 04:45:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7796 [2026-10-22 04:45:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0180 [2026-10-22 04:45:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0692 [2026-10-22 04:45:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4949 [2026-10-22 04:45:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0537 [2026-10-22 04:45:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9746 [2026-10-22 04:45:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3777 [2026-10-22 04:45:38][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,585 | MFU: 1.8% [2026-10-22 04:45:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6790 [2026-10-22 04:45:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4541 [2026-10-22 04:45:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7747 [2026-10-22 04:45:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8350 [2026-10-22 04:45:43][INFO][training.pretrain_engine] [Ph2|Step 551] loss=4.8955 | ppl=133.69 | aux=0.02506 | lr=2.98e-04 | gnorm=0.486 | tok/s=29,294 | MFU=7.0% | step=143179ms | tokens=1.380B | mem=4.2/32.7GB | cost=$45.19/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 13:54 (129.2h) | Phase3: SKIPPED | SFT: Oct 27 18:39 (133.9h) | DPO (Final): Oct 27 19:13 (134.5h) [2026-10-22 04:45:43][INFO][training.pretrain_engine] Starting global step 552 (accumulating 32 micro-batches)... [2026-10-22 04:45:43][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 04:46:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9825 [2026-10-22 04:46:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1593 [2026-10-22 04:46:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8454 [2026-10-22 04:46:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9451 [2026-10-22 04:46:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0178 [2026-10-22 04:46:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9611 [2026-10-22 04:46:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3813 [2026-10-22 04:46:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1361 [2026-10-22 04:46:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.0989 [2026-10-22 04:46:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7811 [2026-10-22 04:46:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6430 [2026-10-22 04:46:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6017 [2026-10-22 04:46:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0788 [2026-10-22 04:46:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7689 [2026-10-22 04:46:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9477 [2026-10-22 04:46:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2461 [2026-10-22 04:47:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8728 [2026-10-22 04:47:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5889 [2026-10-22 04:47:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7323 [2026-10-22 04:47:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3244 [2026-10-22 04:47:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3686 [2026-10-22 04:47:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6561 [2026-10-22 04:47:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.1856 [2026-10-22 04:47:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3205 [2026-10-22 04:47:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1162 [2026-10-22 04:47:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5715 [2026-10-22 04:47:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9124 [2026-10-22 04:47:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7670 [2026-10-22 04:48:01][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,569 | MFU: 1.8% [2026-10-22 04:48:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6976 [2026-10-22 04:48:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8905 [2026-10-22 04:48:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8156 [2026-10-22 04:48:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6032 [2026-10-22 04:48:06][INFO][training.pretrain_engine] [Ph2|Step 552] loss=4.8931 | ppl=133.37 | aux=0.02506 | lr=2.98e-04 | gnorm=0.527 | tok/s=29,295 | MFU=7.0% | step=143177ms | tokens=1.381B | mem=4.2/32.7GB | cost=$45.27/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 13:56 (129.1h) | Phase3: SKIPPED | SFT: Oct 27 18:40 (133.9h) | DPO (Final): Oct 27 19:15 (134.4h) [2026-10-22 04:48:06][INFO][training.pretrain_engine] Starting global step 553 (accumulating 32 micro-batches)... [2026-10-22 04:48:06][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 04:48:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.9382 [2026-10-22 04:48:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8907 [2026-10-22 04:48:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1709 [2026-10-22 04:48:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9466 [2026-10-22 04:48:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7998 [2026-10-22 04:48:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6706 [2026-10-22 04:48:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5154 [2026-10-22 04:48:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6852 [2026-10-22 04:48:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1741 [2026-10-22 04:48:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5629 [2026-10-22 04:48:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1903 [2026-10-22 04:48:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2916 [2026-10-22 04:49:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1358 [2026-10-22 04:49:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3979 [2026-10-22 04:49:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8869 [2026-10-22 04:49:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6089 [2026-10-22 04:49:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8607 [2026-10-22 04:49:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9389 [2026-10-22 04:49:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9791 [2026-10-22 04:49:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7905 [2026-10-22 04:49:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5262 [2026-10-22 04:49:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0643 [2026-10-22 04:49:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8617 [2026-10-22 04:49:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6658 [2026-10-22 04:50:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1513 [2026-10-22 04:50:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0385 [2026-10-22 04:50:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8391 [2026-10-22 04:50:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6827 [2026-10-22 04:50:24][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,594 | MFU: 1.8% [2026-10-22 04:50:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9418 [2026-10-22 04:50:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4675 [2026-10-22 04:50:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3372 [2026-10-22 04:50:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.1226 [2026-10-22 04:50:28][INFO][training.pretrain_engine] [Ph2|Step 553] loss=4.9437 | ppl=140.29 | aux=0.02506 | lr=2.98e-04 | gnorm=0.586 | tok/s=29,296 | MFU=7.0% | step=143172ms | tokens=1.382B | mem=4.2/32.7GB | cost=$45.35/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 13:57 (129.1h) | Phase3: SKIPPED | SFT: Oct 27 18:42 (133.9h) | DPO (Final): Oct 27 19:16 (134.4h) [2026-10-22 04:50:28][INFO][training.pretrain_engine] Starting global step 554 (accumulating 32 micro-batches)... [2026-10-22 04:50:28][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 04:50:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1419 [2026-10-22 04:50:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7091 [2026-10-22 04:50:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6767 [2026-10-22 04:50:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5514 [2026-10-22 04:51:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4600 [2026-10-22 04:51:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9045 [2026-10-22 04:51:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.0295 [2026-10-22 04:51:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1367 [2026-10-22 04:51:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8152 [2026-10-22 04:51:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5203 [2026-10-22 04:51:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.0795 [2026-10-22 04:51:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6016 [2026-10-22 04:51:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9678 [2026-10-22 04:51:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9338 [2026-10-22 04:51:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8173 [2026-10-22 04:51:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8801 [2026-10-22 04:51:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9711 [2026-10-22 04:51:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0465 [2026-10-22 04:51:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7203 [2026-10-22 04:51:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7244 [2026-10-22 04:52:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8988 [2026-10-22 04:52:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0007 [2026-10-22 04:52:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7304 [2026-10-22 04:52:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0738 [2026-10-22 04:52:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1071 [2026-10-22 04:52:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5014 [2026-10-22 04:52:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6737 [2026-10-22 04:52:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6731 [2026-10-22 04:52:46][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,606 | MFU: 1.8% [2026-10-22 04:52:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8609 [2026-10-22 04:52:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2013 [2026-10-22 04:52:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9315 [2026-10-22 04:52:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6986 [2026-10-22 04:52:52][INFO][training.pretrain_engine] [Ph2|Step 554] loss=4.9770 | ppl=145.05 | aux=0.02506 | lr=2.98e-04 | gnorm=0.797 | tok/s=29,297 | MFU=7.0% | step=143165ms | tokens=1.383B | mem=4.2/32.7GB | cost=$45.43/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 13:59 (129.1h) | Phase3: SKIPPED | SFT: Oct 27 18:43 (133.8h) | DPO (Final): Oct 27 19:17 (134.4h) [2026-10-22 04:52:52][INFO][training.pretrain_engine] Starting global step 555 (accumulating 32 micro-batches)... [2026-10-22 04:52:52][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 04:53:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7683 [2026-10-22 04:53:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2544 [2026-10-22 04:53:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9889 [2026-10-22 04:53:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2830 [2026-10-22 04:53:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.6894 [2026-10-22 04:53:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3157 [2026-10-22 04:53:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1371 [2026-10-22 04:53:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1277 [2026-10-22 04:53:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0532 [2026-10-22 04:53:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8445 [2026-10-22 04:53:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0083 [2026-10-22 04:53:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8747 [2026-10-22 04:54:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7431 [2026-10-22 04:54:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5821 [2026-10-22 04:54:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8140 [2026-10-22 04:54:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7373 [2026-10-22 04:54:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5266 [2026-10-22 04:54:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9644 [2026-10-22 04:54:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1315 [2026-10-22 04:54:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3039 [2026-10-22 04:54:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1639 [2026-10-22 04:54:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3692 [2026-10-22 04:54:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4718 [2026-10-22 04:54:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9192 [2026-10-22 04:54:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7311 [2026-10-22 04:54:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8905 [2026-10-22 04:54:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7464 [2026-10-22 04:54:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6534 [2026-10-22 04:55:09][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,628 | MFU: 1.8% [2026-10-22 04:55:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8432 [2026-10-22 04:55:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.0713 [2026-10-22 04:55:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8355 [2026-10-22 04:55:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4716 [2026-10-22 04:55:15][INFO][training.pretrain_engine] [Ph2|Step 555] loss=4.8714 | ppl=130.51 | aux=0.02506 | lr=2.98e-04 | gnorm=0.969 | tok/s=29,296 | MFU=7.0% | step=143170ms | tokens=1.384B | mem=4.2/32.7GB | cost=$45.52/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 14:01 (129.1h) | Phase3: SKIPPED | SFT: Oct 27 18:45 (133.8h) | DPO (Final): Oct 27 19:20 (134.4h) [2026-10-22 04:55:15][INFO][training.pretrain_engine] Starting global step 556 (accumulating 32 micro-batches)... [2026-10-22 04:55:15][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 04:55:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8816 [2026-10-22 04:55:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3936 [2026-10-22 04:55:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4177 [2026-10-22 04:55:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8314 [2026-10-22 04:55:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2306 [2026-10-22 04:55:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0491 [2026-10-22 04:55:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9844 [2026-10-22 04:55:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6960 [2026-10-22 04:56:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3935 [2026-10-22 04:56:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8939 [2026-10-22 04:56:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8410 [2026-10-22 04:56:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9688 [2026-10-22 04:56:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5358 [2026-10-22 04:56:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2851 [2026-10-22 04:56:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1732 [2026-10-22 04:56:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4091 [2026-10-22 04:56:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8579 [2026-10-22 04:56:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5152 [2026-10-22 04:56:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6920 [2026-10-22 04:56:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6473 [2026-10-22 04:57:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.0586 [2026-10-22 04:57:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9397 [2026-10-22 04:57:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0300 [2026-10-22 04:57:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0310 [2026-10-22 04:57:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2067 [2026-10-22 04:57:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3466 [2026-10-22 04:57:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3588 [2026-10-22 04:57:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3700 [2026-10-22 04:57:33][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,606 | MFU: 1.8% [2026-10-22 04:57:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6476 [2026-10-22 04:57:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=3.5853 [2026-10-22 04:57:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5781 [2026-10-22 04:57:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8556 [2026-10-22 04:57:39][INFO][training.pretrain_engine] [Ph2|Step 556] loss=4.8177 | ppl=123.68 | aux=0.02507 | lr=2.98e-04 | gnorm=0.930 | tok/s=29,295 | MFU=7.0% | step=143176ms | tokens=1.385B | mem=4.2/32.7GB | cost=$45.60/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 14:03 (129.1h) | Phase3: SKIPPED | SFT: Oct 27 18:47 (133.8h) | DPO (Final): Oct 27 19:22 (134.4h) [2026-10-22 04:57:39][INFO][training.pretrain_engine] Starting global step 557 (accumulating 32 micro-batches)... [2026-10-22 04:57:39][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 04:57:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1042 [2026-10-22 04:57:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8466 [2026-10-22 04:57:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8005 [2026-10-22 04:57:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9748 [2026-10-22 04:58:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1901 [2026-10-22 04:58:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1471 [2026-10-22 04:58:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0200 [2026-10-22 04:58:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0460 [2026-10-22 04:58:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3902 [2026-10-22 04:58:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9539 [2026-10-22 04:58:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8706 [2026-10-22 04:58:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2736 [2026-10-22 04:58:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1986 [2026-10-22 04:58:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5010 [2026-10-22 04:58:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4342 [2026-10-22 04:58:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2482 [2026-10-22 04:59:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9998 [2026-10-22 04:59:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1883 [2026-10-22 04:59:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9526 [2026-10-22 04:59:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0861 [2026-10-22 04:59:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7659 [2026-10-22 04:59:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0372 [2026-10-22 04:59:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.2136 [2026-10-22 04:59:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9828 [2026-10-22 04:59:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4971 [2026-10-22 04:59:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5792 [2026-10-22 04:59:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5221 [2026-10-22 04:59:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4029 [2026-10-22 04:59:57][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,575 | MFU: 1.8% [2026-10-22 05:00:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3574 [2026-10-22 05:00:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2970 [2026-10-22 05:00:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6946 [2026-10-22 05:00:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9480 [2026-10-22 05:00:03][INFO][training.pretrain_engine] [Ph2|Step 557] loss=4.9781 | ppl=145.20 | aux=0.02507 | lr=2.98e-04 | gnorm=0.902 | tok/s=29,293 | MFU=7.0% | step=143186ms | tokens=1.386B | mem=4.2/32.7GB | cost=$45.68/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 14:05 (129.1h) | Phase3: SKIPPED | SFT: Oct 27 18:50 (133.8h) | DPO (Final): Oct 27 19:24 (134.4h) [2026-10-22 05:00:03][INFO][training.pretrain_engine] Starting global step 558 (accumulating 32 micro-batches)... [2026-10-22 05:00:03][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 05:00:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0954 [2026-10-22 05:00:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5418 [2026-10-22 05:00:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8510 [2026-10-22 05:00:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0766 [2026-10-22 05:00:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9630 [2026-10-22 05:00:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6962 [2026-10-22 05:00:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6241 [2026-10-22 05:00:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5516 [2026-10-22 05:00:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2254 [2026-10-22 05:00:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=3.9984 [2026-10-22 05:00:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=3.7083 [2026-10-22 05:00:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8924 [2026-10-22 05:01:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0330 [2026-10-22 05:01:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0905 [2026-10-22 05:01:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2337 [2026-10-22 05:01:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0876 [2026-10-22 05:01:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9058 [2026-10-22 05:01:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9235 [2026-10-22 05:01:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.2349 [2026-10-22 05:01:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9560 [2026-10-22 05:01:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0911 [2026-10-22 05:01:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4107 [2026-10-22 05:01:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1114 [2026-10-22 05:01:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0041 [2026-10-22 05:02:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9192 [2026-10-22 05:02:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0910 [2026-10-22 05:02:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0735 [2026-10-22 05:02:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1654 [2026-10-22 05:02:21][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,584 | MFU: 1.8% [2026-10-22 05:02:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9331 [2026-10-22 05:02:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0466 [2026-10-22 05:02:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4634 [2026-10-22 05:02:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7058 [2026-10-22 05:02:27][INFO][training.pretrain_engine] [Ph2|Step 558] loss=4.9182 | ppl=136.75 | aux=0.02506 | lr=2.98e-04 | gnorm=0.730 | tok/s=29,292 | MFU=7.0% | step=143189ms | tokens=1.387B | mem=4.2/32.7GB | cost=$45.76/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 14:07 (129.1h) | Phase3: SKIPPED | SFT: Oct 27 18:52 (133.8h) | DPO (Final): Oct 27 19:26 (134.4h) [2026-10-22 05:02:30][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step558 | Tokens: 1.387B | Spent: $45.77 | Remaining: $954.23 [2026-10-22 05:02:30][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step558 [2026-10-22 05:02:30][INFO][training.pretrain_engine] Starting global step 559 (accumulating 32 micro-batches)... [2026-10-22 05:02:30][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 05:02:30][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-22 05:02:30][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-22 05:02:30][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-22 05:02:30][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step558 to Phase-Technologies/forge-500m-pretrain (path: phase2_step558)... [2026-10-22 05:02:30][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-22 05:02:39][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-22 05:02:39][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step558 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-22 05:02:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5853 [2026-10-22 05:02:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8217 [2026-10-22 05:02:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6161 [2026-10-22 05:02:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6319 [2026-10-22 05:03:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3126 [2026-10-22 05:03:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3373 [2026-10-22 05:03:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1651 [2026-10-22 05:03:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0194 [2026-10-22 05:03:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9455 [2026-10-22 05:03:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9668 [2026-10-22 05:03:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1298 [2026-10-22 05:03:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9206 [2026-10-22 05:03:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2951 [2026-10-22 05:03:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4536 [2026-10-22 05:03:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5501 [2026-10-22 05:03:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8534 [2026-10-22 05:03:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8030 [2026-10-22 05:03:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9101 [2026-10-22 05:03:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1305 [2026-10-22 05:03:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4445 [2026-10-22 05:04:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9743 [2026-10-22 05:04:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6501 [2026-10-22 05:04:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0095 [2026-10-22 05:04:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6159 [2026-10-22 05:04:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2588 [2026-10-22 05:04:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1712 [2026-10-22 05:04:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1944 [2026-10-22 05:04:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4477 [2026-10-22 05:04:48][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,605 | MFU: 1.8% [2026-10-22 05:04:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1146 [2026-10-22 05:04:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3186 [2026-10-22 05:04:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5271 [2026-10-22 05:04:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0033 [2026-10-22 05:04:52][INFO][training.pretrain_engine] [Ph2|Step 559] loss=4.8383 | ppl=126.25 | aux=0.02506 | lr=2.98e-04 | gnorm=0.594 | tok/s=29,294 | MFU=7.0% | step=143181ms | tokens=1.388B | mem=4.2/32.7GB | cost=$45.85/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 14:09 (129.1h) | Phase3: SKIPPED | SFT: Oct 27 18:53 (133.8h) | DPO (Final): Oct 27 19:27 (134.4h) [2026-10-22 05:04:52][INFO][training.pretrain_engine] Starting global step 560 (accumulating 32 micro-batches)... [2026-10-22 05:04:52][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 05:05:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.6341 [2026-10-22 05:05:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.9347 [2026-10-22 05:05:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3100 [2026-10-22 05:05:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6132 [2026-10-22 05:05:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7896 [2026-10-22 05:05:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4390 [2026-10-22 05:05:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2961 [2026-10-22 05:05:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3964 [2026-10-22 05:05:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7057 [2026-10-22 05:05:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0280 [2026-10-22 05:05:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0186 [2026-10-22 05:05:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3645 [2026-10-22 05:06:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3286 [2026-10-22 05:06:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6043 [2026-10-22 05:06:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0126 [2026-10-22 05:06:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9457 [2026-10-22 05:06:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1065 [2026-10-22 05:06:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8346 [2026-10-22 05:06:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7524 [2026-10-22 05:06:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8977 [2026-10-22 05:06:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7540 [2026-10-22 05:06:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7530 [2026-10-22 05:06:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9819 [2026-10-22 05:06:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4949 [2026-10-22 05:06:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5857 [2026-10-22 05:06:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8948 [2026-10-22 05:06:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4300 [2026-10-22 05:06:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8545 [2026-10-22 05:07:11][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,571 | MFU: 1.8% [2026-10-22 05:07:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9306 [2026-10-22 05:07:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0298 [2026-10-22 05:07:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7553 [2026-10-22 05:07:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1492 [2026-10-22 05:07:16][INFO][training.pretrain_engine] [Ph2|Step 560] loss=4.9018 | ppl=134.53 | aux=0.02507 | lr=2.98e-04 | gnorm=0.688 | tok/s=29,291 | MFU=7.0% | step=143194ms | tokens=1.389B | mem=4.2/32.7GB | cost=$45.93/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 14:11 (129.1h) | Phase3: SKIPPED | SFT: Oct 27 18:56 (133.8h) | DPO (Final): Oct 27 19:30 (134.4h) [2026-10-22 05:07:16][INFO][training.pretrain_engine] Starting global step 561 (accumulating 32 micro-batches)... [2026-10-22 05:07:16][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 05:07:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3393 [2026-10-22 05:07:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8676 [2026-10-22 05:07:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1256 [2026-10-22 05:07:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8301 [2026-10-22 05:07:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5912 [2026-10-22 05:07:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1773 [2026-10-22 05:07:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8802 [2026-10-22 05:07:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1728 [2026-10-22 05:08:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1642 [2026-10-22 05:08:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6479 [2026-10-22 05:08:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2380 [2026-10-22 05:08:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1972 [2026-10-22 05:08:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9816 [2026-10-22 05:08:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6285 [2026-10-22 05:08:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0705 [2026-10-22 05:08:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6415 [2026-10-22 05:08:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6022 [2026-10-22 05:08:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7096 [2026-10-22 05:08:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0769 [2026-10-22 05:08:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6620 [2026-10-22 05:09:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6427 [2026-10-22 05:09:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=2.9873 [2026-10-22 05:09:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3029 [2026-10-22 05:09:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=3.9727 [2026-10-22 05:09:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3056 [2026-10-22 05:09:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.1739 [2026-10-22 05:09:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7210 [2026-10-22 05:09:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=3.8518 [2026-10-22 05:09:34][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,582 | MFU: 1.8% [2026-10-22 05:09:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0255 [2026-10-22 05:09:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3363 [2026-10-22 05:09:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8622 [2026-10-22 05:09:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6735 [2026-10-22 05:09:40][INFO][training.pretrain_engine] [Ph2|Step 561] loss=4.9069 | ppl=135.22 | aux=0.02505 | lr=2.97e-04 | gnorm=0.664 | tok/s=29,290 | MFU=7.0% | step=143201ms | tokens=1.390B | mem=4.2/32.7GB | cost=$46.01/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 14:13 (129.1h) | Phase3: SKIPPED | SFT: Oct 27 18:58 (133.8h) | DPO (Final): Oct 27 19:32 (134.4h) [2026-10-22 05:09:40][INFO][training.pretrain_engine] Starting global step 562 (accumulating 32 micro-batches)... [2026-10-22 05:09:40][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 05:09:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0018 [2026-10-22 05:09:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6725 [2026-10-22 05:09:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9372 [2026-10-22 05:09:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2661 [2026-10-22 05:10:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7797 [2026-10-22 05:10:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9335 [2026-10-22 05:10:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9561 [2026-10-22 05:10:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1095 [2026-10-22 05:10:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3797 [2026-10-22 05:10:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8358 [2026-10-22 05:10:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9564 [2026-10-22 05:10:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3430 [2026-10-22 05:10:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5887 [2026-10-22 05:10:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3153 [2026-10-22 05:10:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6985 [2026-10-22 05:10:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6184 [2026-10-22 05:11:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9697 [2026-10-22 05:11:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5835 [2026-10-22 05:11:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7156 [2026-10-22 05:11:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5784 [2026-10-22 05:11:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6233 [2026-10-22 05:11:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8570 [2026-10-22 05:11:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5852 [2026-10-22 05:11:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9117 [2026-10-22 05:11:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6726 [2026-10-22 05:11:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4536 [2026-10-22 05:11:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7028 [2026-10-22 05:11:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.1638 [2026-10-22 05:11:58][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,586 | MFU: 1.8% [2026-10-22 05:12:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4174 [2026-10-22 05:12:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0121 [2026-10-22 05:12:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0679 [2026-10-22 05:12:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3953 [2026-10-22 05:12:04][INFO][training.pretrain_engine] [Ph2|Step 562] loss=4.7631 | ppl=117.10 | aux=0.02506 | lr=2.97e-04 | gnorm=0.625 | tok/s=29,289 | MFU=7.0% | step=143205ms | tokens=1.391B | mem=4.2/32.7GB | cost=$46.09/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 14:15 (129.1h) | Phase3: SKIPPED | SFT: Oct 27 19:00 (133.8h) | DPO (Final): Oct 27 19:34 (134.4h) [2026-10-22 05:12:04][INFO][training.pretrain_engine] Starting global step 563 (accumulating 32 micro-batches)... [2026-10-22 05:12:04][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 05:12:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4933 [2026-10-22 05:12:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9224 [2026-10-22 05:12:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7237 [2026-10-22 05:12:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4426 [2026-10-22 05:12:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8437 [2026-10-22 05:12:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7682 [2026-10-22 05:12:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6443 [2026-10-22 05:12:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2884 [2026-10-22 05:12:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9349 [2026-10-22 05:12:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0596 [2026-10-22 05:12:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3985 [2026-10-22 05:12:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2316 [2026-10-22 05:13:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1707 [2026-10-22 05:13:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2631 [2026-10-22 05:13:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9812 [2026-10-22 05:13:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2936 [2026-10-22 05:13:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3429 [2026-10-22 05:13:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6083 [2026-10-22 05:13:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0561 [2026-10-22 05:13:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7122 [2026-10-22 05:13:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3340 [2026-10-22 05:13:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3272 [2026-10-22 05:13:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8239 [2026-10-22 05:13:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1179 [2026-10-22 05:14:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3456 [2026-10-22 05:14:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9187 [2026-10-22 05:14:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2332 [2026-10-22 05:14:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3013 [2026-10-22 05:14:23][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,571 | MFU: 1.8% [2026-10-22 05:14:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2148 [2026-10-22 05:14:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5406 [2026-10-22 05:14:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8604 [2026-10-22 05:14:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4541 [2026-10-22 05:14:28][INFO][training.pretrain_engine] [Ph2|Step 563] loss=4.9186 | ppl=136.81 | aux=0.02506 | lr=2.97e-04 | gnorm=0.621 | tok/s=29,287 | MFU=7.0% | step=143212ms | tokens=1.393B | mem=4.2/32.7GB | cost=$46.18/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 14:18 (129.1h) | Phase3: SKIPPED | SFT: Oct 27 19:02 (133.8h) | DPO (Final): Oct 27 19:36 (134.4h) [2026-10-22 05:14:28][INFO][training.pretrain_engine] Starting global step 564 (accumulating 32 micro-batches)... [2026-10-22 05:14:28][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 05:14:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.8987 [2026-10-22 05:14:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4287 [2026-10-22 05:14:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7007 [2026-10-22 05:14:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7778 [2026-10-22 05:15:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9017 [2026-10-22 05:15:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7858 [2026-10-22 05:15:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7017 [2026-10-22 05:15:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8256 [2026-10-22 05:15:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2313 [2026-10-22 05:15:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3584 [2026-10-22 05:15:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7318 [2026-10-22 05:15:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8327 [2026-10-22 05:15:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6990 [2026-10-22 05:15:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8559 [2026-10-22 05:15:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4968 [2026-10-22 05:15:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6001 [2026-10-22 05:15:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5059 [2026-10-22 05:15:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4787 [2026-10-22 05:15:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9321 [2026-10-22 05:15:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4990 [2026-10-22 05:16:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3707 [2026-10-22 05:16:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2216 [2026-10-22 05:16:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1011 [2026-10-22 05:16:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0614 [2026-10-22 05:16:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8579 [2026-10-22 05:16:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3553 [2026-10-22 05:16:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3130 [2026-10-22 05:16:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2016 [2026-10-22 05:16:46][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,580 | MFU: 1.8% [2026-10-22 05:16:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=3.6495 [2026-10-22 05:16:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1062 [2026-10-22 05:16:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.1241 [2026-10-22 05:16:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2296 [2026-10-22 05:16:51][INFO][training.pretrain_engine] [Ph2|Step 564] loss=4.7659 | ppl=117.44 | aux=0.02505 | lr=2.97e-04 | gnorm=0.527 | tok/s=29,286 | MFU=7.0% | step=143220ms | tokens=1.394B | mem=4.2/32.7GB | cost=$46.26/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 14:20 (129.1h) | Phase3: SKIPPED | SFT: Oct 27 19:04 (133.8h) | DPO (Final): Oct 27 19:39 (134.4h) [2026-10-22 05:16:51][INFO][training.pretrain_engine] Starting global step 565 (accumulating 32 micro-batches)... [2026-10-22 05:16:51][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 05:17:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0991 [2026-10-22 05:17:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9996 [2026-10-22 05:17:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0040 [2026-10-22 05:17:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8213 [2026-10-22 05:17:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1194 [2026-10-22 05:17:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9825 [2026-10-22 05:17:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1486 [2026-10-22 05:17:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8615 [2026-10-22 05:17:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4377 [2026-10-22 05:17:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5572 [2026-10-22 05:17:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7523 [2026-10-22 05:17:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7225 [2026-10-22 05:18:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6542 [2026-10-22 05:18:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1830 [2026-10-22 05:18:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1140 [2026-10-22 05:18:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0033 [2026-10-22 05:18:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9552 [2026-10-22 05:18:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7676 [2026-10-22 05:18:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5099 [2026-10-22 05:18:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8424 [2026-10-22 05:18:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4000 [2026-10-22 05:18:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9742 [2026-10-22 05:18:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1258 [2026-10-22 05:18:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2838 [2026-10-22 05:18:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4063 [2026-10-22 05:18:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3051 [2026-10-22 05:18:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2955 [2026-10-22 05:18:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8913 [2026-10-22 05:19:10][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,575 | MFU: 1.8% [2026-10-22 05:19:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0379 [2026-10-22 05:19:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9463 [2026-10-22 05:19:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4463 [2026-10-22 05:19:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1936 [2026-10-22 05:19:15][INFO][training.pretrain_engine] [Ph2|Step 565] loss=4.9450 | ppl=140.47 | aux=0.02505 | lr=2.97e-04 | gnorm=0.625 | tok/s=29,283 | MFU=7.0% | step=143231ms | tokens=1.395B | mem=4.2/32.7GB | cost=$46.34/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 14:22 (129.1h) | Phase3: SKIPPED | SFT: Oct 27 19:07 (133.8h) | DPO (Final): Oct 27 19:41 (134.4h) [2026-10-22 05:19:15][INFO][training.pretrain_engine] Starting global step 566 (accumulating 32 micro-batches)... [2026-10-22 05:19:15][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 05:19:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6549 [2026-10-22 05:19:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7177 [2026-10-22 05:19:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7638 [2026-10-22 05:19:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7298 [2026-10-22 05:19:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1588 [2026-10-22 05:19:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6068 [2026-10-22 05:19:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6104 [2026-10-22 05:19:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3572 [2026-10-22 05:20:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7726 [2026-10-22 05:20:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1693 [2026-10-22 05:20:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0154 [2026-10-22 05:20:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8379 [2026-10-22 05:20:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8429 [2026-10-22 05:20:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.1441 [2026-10-22 05:20:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6610 [2026-10-22 05:20:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8100 [2026-10-22 05:20:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1409 [2026-10-22 05:20:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8777 [2026-10-22 05:20:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9355 [2026-10-22 05:20:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7608 [2026-10-22 05:21:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8377 [2026-10-22 05:21:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3984 [2026-10-22 05:21:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0899 [2026-10-22 05:21:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5461 [2026-10-22 05:21:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0205 [2026-10-22 05:21:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6949 [2026-10-22 05:21:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9081 [2026-10-22 05:21:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8550 [2026-10-22 05:21:34][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,592 | MFU: 1.8% [2026-10-22 05:21:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0106 [2026-10-22 05:21:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0206 [2026-10-22 05:21:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6528 [2026-10-22 05:21:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1786 [2026-10-22 05:21:39][INFO][training.pretrain_engine] [Ph2|Step 566] loss=4.9154 | ppl=136.38 | aux=0.02505 | lr=2.97e-04 | gnorm=0.648 | tok/s=29,282 | MFU=7.0% | step=143239ms | tokens=1.396B | mem=4.2/32.7GB | cost=$46.42/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 14:24 (129.1h) | Phase3: SKIPPED | SFT: Oct 27 19:09 (133.8h) | DPO (Final): Oct 27 19:43 (134.4h) [2026-10-22 05:21:39][INFO][training.pretrain_engine] Starting global step 567 (accumulating 32 micro-batches)... [2026-10-22 05:21:39][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 05:21:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0007 [2026-10-22 05:21:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9693 [2026-10-22 05:21:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1160 [2026-10-22 05:21:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0323 [2026-10-22 05:22:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4367 [2026-10-22 05:22:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3412 [2026-10-22 05:22:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4876 [2026-10-22 05:22:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0788 [2026-10-22 05:22:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1217 [2026-10-22 05:22:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7263 [2026-10-22 05:22:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3692 [2026-10-22 05:22:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0527 [2026-10-22 05:22:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8802 [2026-10-22 05:22:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8745 [2026-10-22 05:22:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6905 [2026-10-22 05:22:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6388 [2026-10-22 05:23:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8472 [2026-10-22 05:23:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7164 [2026-10-22 05:23:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5166 [2026-10-22 05:23:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7291 [2026-10-22 05:23:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0238 [2026-10-22 05:23:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0407 [2026-10-22 05:23:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9721 [2026-10-22 05:23:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9898 [2026-10-22 05:23:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3535 [2026-10-22 05:23:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4940 [2026-10-22 05:23:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3550 [2026-10-22 05:23:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1759 [2026-10-22 05:23:57][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,583 | MFU: 1.8% [2026-10-22 05:24:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5590 [2026-10-22 05:24:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9963 [2026-10-22 05:24:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8752 [2026-10-22 05:24:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0435 [2026-10-22 05:24:02][INFO][training.pretrain_engine] [Ph2|Step 567] loss=4.8822 | ppl=131.93 | aux=0.02505 | lr=2.97e-04 | gnorm=0.578 | tok/s=29,281 | MFU=7.0% | step=143243ms | tokens=1.397B | mem=4.2/32.7GB | cost=$46.51/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 14:26 (129.0h) | Phase3: SKIPPED | SFT: Oct 27 19:11 (133.8h) | DPO (Final): Oct 27 19:45 (134.4h) [2026-10-22 05:24:06][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step567 | Tokens: 1.397B | Spent: $46.51 | Remaining: $953.49 [2026-10-22 05:24:06][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step567 [2026-10-22 05:24:06][INFO][training.pretrain_engine] Starting global step 568 (accumulating 32 micro-batches)... [2026-10-22 05:24:06][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 05:24:06][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-22 05:24:06][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-22 05:24:06][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-22 05:24:06][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step567 to Phase-Technologies/forge-500m-pretrain (path: phase2_step567)... [2026-10-22 05:24:06][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-22 05:24:15][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-22 05:24:15][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step567 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-22 05:24:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8671 [2026-10-22 05:24:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6582 [2026-10-22 05:24:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6316 [2026-10-22 05:24:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6773 [2026-10-22 05:24:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2965 [2026-10-22 05:24:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2281 [2026-10-22 05:24:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5017 [2026-10-22 05:24:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2789 [2026-10-22 05:24:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2275 [2026-10-22 05:24:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7068 [2026-10-22 05:24:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3281 [2026-10-22 05:24:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3981 [2026-10-22 05:25:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5099 [2026-10-22 05:25:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8999 [2026-10-22 05:25:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.1389 [2026-10-22 05:25:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7314 [2026-10-22 05:25:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8603 [2026-10-22 05:25:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6687 [2026-10-22 05:25:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5542 [2026-10-22 05:25:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6462 [2026-10-22 05:25:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8536 [2026-10-22 05:25:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6685 [2026-10-22 05:25:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7312 [2026-10-22 05:25:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7444 [2026-10-22 05:26:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5484 [2026-10-22 05:26:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5564 [2026-10-22 05:26:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6171 [2026-10-22 05:26:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8320 [2026-10-22 05:26:24][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,577 | MFU: 1.8% [2026-10-22 05:26:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8791 [2026-10-22 05:26:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9162 [2026-10-22 05:26:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1527 [2026-10-22 05:26:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0937 [2026-10-22 05:26:28][INFO][training.pretrain_engine] [Ph2|Step 568] loss=4.7784 | ppl=118.91 | aux=0.02507 | lr=2.97e-04 | gnorm=0.566 | tok/s=29,283 | MFU=7.0% | step=143234ms | tokens=1.398B | mem=4.2/32.7GB | cost=$46.59/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 14:28 (129.0h) | Phase3: SKIPPED | SFT: Oct 27 19:12 (133.8h) | DPO (Final): Oct 27 19:47 (134.3h) [2026-10-22 05:26:28][INFO][training.pretrain_engine] Starting global step 569 (accumulating 32 micro-batches)... [2026-10-22 05:26:28][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 05:26:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0652 [2026-10-22 05:26:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5614 [2026-10-22 05:26:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4639 [2026-10-22 05:26:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7526 [2026-10-22 05:27:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9760 [2026-10-22 05:27:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9305 [2026-10-22 05:27:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7175 [2026-10-22 05:27:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0501 [2026-10-22 05:27:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0253 [2026-10-22 05:27:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1761 [2026-10-22 05:27:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8773 [2026-10-22 05:27:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3696 [2026-10-22 05:27:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9679 [2026-10-22 05:27:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7337 [2026-10-22 05:27:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3250 [2026-10-22 05:27:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1448 [2026-10-22 05:27:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4174 [2026-10-22 05:27:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5581 [2026-10-22 05:27:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4970 [2026-10-22 05:27:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4777 [2026-10-22 05:28:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4785 [2026-10-22 05:28:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=3.8424 [2026-10-22 05:28:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=3.5300 [2026-10-22 05:28:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4099 [2026-10-22 05:28:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.1698 [2026-10-22 05:28:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4963 [2026-10-22 05:28:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9065 [2026-10-22 05:28:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5590 [2026-10-22 05:28:47][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,569 | MFU: 1.8% [2026-10-22 05:28:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7739 [2026-10-22 05:28:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0883 [2026-10-22 05:28:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0465 [2026-10-22 05:28:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5065 [2026-10-22 05:28:52][INFO][training.pretrain_engine] [Ph2|Step 569] loss=4.9086 | ppl=135.45 | aux=0.02507 | lr=2.97e-04 | gnorm=0.543 | tok/s=29,281 | MFU=7.0% | step=143242ms | tokens=1.399B | mem=4.2/32.7GB | cost=$46.67/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 14:30 (129.0h) | Phase3: SKIPPED | SFT: Oct 27 19:15 (133.8h) | DPO (Final): Oct 27 19:49 (134.3h) [2026-10-22 05:28:52][INFO][training.pretrain_engine] Starting global step 570 (accumulating 32 micro-batches)... [2026-10-22 05:28:52][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 05:29:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7121 [2026-10-22 05:29:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3051 [2026-10-22 05:29:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6553 [2026-10-22 05:29:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9097 [2026-10-22 05:29:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0201 [2026-10-22 05:29:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4387 [2026-10-22 05:29:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0068 [2026-10-22 05:29:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3638 [2026-10-22 05:29:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0128 [2026-10-22 05:29:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0026 [2026-10-22 05:29:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9673 [2026-10-22 05:29:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9728 [2026-10-22 05:30:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=3.3310 [2026-10-22 05:30:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.1374 [2026-10-22 05:30:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5784 [2026-10-22 05:30:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3008 [2026-10-22 05:30:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7817 [2026-10-22 05:30:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7481 [2026-10-22 05:30:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8016 [2026-10-22 05:30:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0328 [2026-10-22 05:30:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7438 [2026-10-22 05:30:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6912 [2026-10-22 05:30:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7317 [2026-10-22 05:30:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9134 [2026-10-22 05:30:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4087 [2026-10-22 05:30:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.3943 [2026-10-22 05:30:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8867 [2026-10-22 05:30:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5207 [2026-10-22 05:31:10][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,592 | MFU: 1.8% [2026-10-22 05:31:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3475 [2026-10-22 05:31:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.0591 [2026-10-22 05:31:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2642 [2026-10-22 05:31:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2677 [2026-10-22 05:31:15][INFO][training.pretrain_engine] [Ph2|Step 570] loss=4.8997 | ppl=134.25 | aux=0.02506 | lr=2.97e-04 | gnorm=0.500 | tok/s=29,283 | MFU=7.0% | step=143235ms | tokens=1.400B | mem=4.2/32.7GB | cost=$46.75/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 14:31 (129.0h) | Phase3: SKIPPED | SFT: Oct 27 19:16 (133.8h) | DPO (Final): Oct 27 19:50 (134.3h) [2026-10-22 05:31:15][INFO][training.pretrain_engine] Starting global step 571 (accumulating 32 micro-batches)... [2026-10-22 05:31:15][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 05:31:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0193 [2026-10-22 05:31:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2188 [2026-10-22 05:31:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1743 [2026-10-22 05:31:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5182 [2026-10-22 05:31:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9171 [2026-10-22 05:31:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4066 [2026-10-22 05:31:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2370 [2026-10-22 05:31:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1164 [2026-10-22 05:32:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6709 [2026-10-22 05:32:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7639 [2026-10-22 05:32:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7156 [2026-10-22 05:32:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4264 [2026-10-22 05:32:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8593 [2026-10-22 05:32:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5958 [2026-10-22 05:32:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8620 [2026-10-22 05:32:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1317 [2026-10-22 05:32:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7924 [2026-10-22 05:32:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9631 [2026-10-22 05:32:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7646 [2026-10-22 05:32:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7964 [2026-10-22 05:33:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3792 [2026-10-22 05:33:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7667 [2026-10-22 05:33:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4584 [2026-10-22 05:33:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3106 [2026-10-22 05:33:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8410 [2026-10-22 05:33:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5976 [2026-10-22 05:33:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0360 [2026-10-22 05:33:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0786 [2026-10-22 05:33:33][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,587 | MFU: 1.8% [2026-10-22 05:33:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5253 [2026-10-22 05:33:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0907 [2026-10-22 05:33:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8535 [2026-10-22 05:33:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9825 [2026-10-22 05:33:38][INFO][training.pretrain_engine] [Ph2|Step 571] loss=4.8507 | ppl=127.83 | aux=0.02506 | lr=2.97e-04 | gnorm=0.459 | tok/s=29,284 | MFU=7.0% | step=143227ms | tokens=1.401B | mem=4.2/32.7GB | cost=$46.83/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 14:33 (129.0h) | Phase3: SKIPPED | SFT: Oct 27 19:17 (133.7h) | DPO (Final): Oct 27 19:52 (134.3h) [2026-10-22 05:33:38][INFO][training.pretrain_engine] Starting global step 572 (accumulating 32 micro-batches)... [2026-10-22 05:33:38][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 05:33:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0904 [2026-10-22 05:33:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6612 [2026-10-22 05:33:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6252 [2026-10-22 05:33:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1785 [2026-10-22 05:34:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8646 [2026-10-22 05:34:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6250 [2026-10-22 05:34:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3694 [2026-10-22 05:34:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6834 [2026-10-22 05:34:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5986 [2026-10-22 05:34:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7128 [2026-10-22 05:34:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5123 [2026-10-22 05:34:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4730 [2026-10-22 05:34:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7055 [2026-10-22 05:34:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7937 [2026-10-22 05:34:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5965 [2026-10-22 05:34:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6221 [2026-10-22 05:35:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1072 [2026-10-22 05:35:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7431 [2026-10-22 05:35:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8747 [2026-10-22 05:35:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0764 [2026-10-22 05:35:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2077 [2026-10-22 05:35:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7511 [2026-10-22 05:35:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1126 [2026-10-22 05:35:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2397 [2026-10-22 05:35:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4933 [2026-10-22 05:35:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.5645 [2026-10-22 05:35:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0475 [2026-10-22 05:35:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4711 [2026-10-22 05:35:57][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,570 | MFU: 1.8% [2026-10-22 05:36:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5558 [2026-10-22 05:36:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2709 [2026-10-22 05:36:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5146 [2026-10-22 05:36:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0228 [2026-10-22 05:36:02][INFO][training.pretrain_engine] [Ph2|Step 572] loss=4.8303 | ppl=125.25 | aux=0.02506 | lr=2.97e-04 | gnorm=0.562 | tok/s=29,283 | MFU=7.0% | step=143235ms | tokens=1.402B | mem=4.2/32.7GB | cost=$46.92/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 14:35 (129.0h) | Phase3: SKIPPED | SFT: Oct 27 19:20 (133.7h) | DPO (Final): Oct 27 19:54 (134.3h) [2026-10-22 05:36:02][INFO][training.pretrain_engine] Starting global step 573 (accumulating 32 micro-batches)... [2026-10-22 05:36:02][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 05:36:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6466 [2026-10-22 05:36:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1062 [2026-10-22 05:36:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6742 [2026-10-22 05:36:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1891 [2026-10-22 05:36:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0173 [2026-10-22 05:36:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6045 [2026-10-22 05:36:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8509 [2026-10-22 05:36:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8375 [2026-10-22 05:36:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.1979 [2026-10-22 05:36:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5734 [2026-10-22 05:36:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5497 [2026-10-22 05:36:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3567 [2026-10-22 05:37:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0478 [2026-10-22 05:37:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9791 [2026-10-22 05:37:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8084 [2026-10-22 05:37:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0829 [2026-10-22 05:37:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7341 [2026-10-22 05:37:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=3.5846 [2026-10-22 05:37:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6486 [2026-10-22 05:37:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=3.5980 [2026-10-22 05:37:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4066 [2026-10-22 05:37:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1672 [2026-10-22 05:37:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0407 [2026-10-22 05:37:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1293 [2026-10-22 05:38:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5130 [2026-10-22 05:38:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8092 [2026-10-22 05:38:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9122 [2026-10-22 05:38:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9634 [2026-10-22 05:38:21][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,565 | MFU: 1.8% [2026-10-22 05:38:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7736 [2026-10-22 05:38:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5628 [2026-10-22 05:38:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3032 [2026-10-22 05:38:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6178 [2026-10-22 05:38:26][INFO][training.pretrain_engine] [Ph2|Step 573] loss=4.8713 | ppl=130.48 | aux=0.02506 | lr=2.97e-04 | gnorm=0.508 | tok/s=29,282 | MFU=7.0% | step=143239ms | tokens=1.403B | mem=4.2/32.7GB | cost=$47.00/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 14:37 (129.0h) | Phase3: SKIPPED | SFT: Oct 27 19:22 (133.7h) | DPO (Final): Oct 27 19:56 (134.3h) [2026-10-22 05:38:26][INFO][training.pretrain_engine] Starting global step 574 (accumulating 32 micro-batches)... [2026-10-22 05:38:26][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 05:38:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7949 [2026-10-22 05:38:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1446 [2026-10-22 05:38:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9891 [2026-10-22 05:38:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3635 [2026-10-22 05:39:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5094 [2026-10-22 05:39:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8495 [2026-10-22 05:39:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5849 [2026-10-22 05:39:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0446 [2026-10-22 05:39:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4907 [2026-10-22 05:39:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8216 [2026-10-22 05:39:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2609 [2026-10-22 05:39:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8097 [2026-10-22 05:39:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8524 [2026-10-22 05:39:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5794 [2026-10-22 05:39:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1011 [2026-10-22 05:39:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5797 [2026-10-22 05:39:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4185 [2026-10-22 05:39:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3431 [2026-10-22 05:39:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3456 [2026-10-22 05:39:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2153 [2026-10-22 05:40:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1174 [2026-10-22 05:40:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3233 [2026-10-22 05:40:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2503 [2026-10-22 05:40:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0985 [2026-10-22 05:40:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8786 [2026-10-22 05:40:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1179 [2026-10-22 05:40:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9742 [2026-10-22 05:40:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9654 [2026-10-22 05:40:44][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,597 | MFU: 1.8% [2026-10-22 05:40:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1704 [2026-10-22 05:40:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6460 [2026-10-22 05:40:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8653 [2026-10-22 05:40:49][INFO][training.pretrain_engine] [Ph2|Step 574] loss=4.9310 | ppl=138.52 | aux=0.02505 | lr=2.97e-04 | gnorm=0.555 | tok/s=29,281 | MFU=7.0% | step=143245ms | tokens=1.404B | mem=4.2/32.7GB | cost=$47.08/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 14:39 (129.0h) | Phase3: SKIPPED | SFT: Oct 27 19:24 (133.7h) | DPO (Final): Oct 27 19:58 (134.3h) [2026-10-22 05:40:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9103 [2026-10-22 05:40:49][INFO][training.pretrain_engine] Starting global step 575 (accumulating 32 micro-batches)... [2026-10-22 05:40:49][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 05:41:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1596 [2026-10-22 05:41:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.1741 [2026-10-22 05:41:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7636 [2026-10-22 05:41:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4469 [2026-10-22 05:41:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9297 [2026-10-22 05:41:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5403 [2026-10-22 05:41:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7033 [2026-10-22 05:41:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8257 [2026-10-22 05:41:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6383 [2026-10-22 05:41:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8971 [2026-10-22 05:41:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0379 [2026-10-22 05:41:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9964 [2026-10-22 05:42:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7854 [2026-10-22 05:42:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9267 [2026-10-22 05:42:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2011 [2026-10-22 05:42:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5662 [2026-10-22 05:42:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9397 [2026-10-22 05:42:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1021 [2026-10-22 05:42:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0176 [2026-10-22 05:42:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6961 [2026-10-22 05:42:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8767 [2026-10-22 05:42:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9618 [2026-10-22 05:42:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7081 [2026-10-22 05:42:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0512 [2026-10-22 05:42:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0095 [2026-10-22 05:42:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9943 [2026-10-22 05:42:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0129 [2026-10-22 05:42:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3686 [2026-10-22 05:43:07][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,601 | MFU: 1.8% [2026-10-22 05:43:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0967 [2026-10-22 05:43:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8391 [2026-10-22 05:43:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8337 [2026-10-22 05:43:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8154 [2026-10-22 05:43:13][INFO][training.pretrain_engine] [Ph2|Step 575] loss=4.7095 | ppl=111.00 | aux=0.02508 | lr=2.97e-04 | gnorm=0.547 | tok/s=29,280 | MFU=7.0% | step=143246ms | tokens=1.405B | mem=4.2/32.7GB | cost=$47.16/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 14:41 (129.0h) | Phase3: SKIPPED | SFT: Oct 27 19:26 (133.7h) | DPO (Final): Oct 27 20:00 (134.3h) [2026-10-22 05:43:13][INFO][training.pretrain_engine] Starting global step 576 (accumulating 32 micro-batches)... [2026-10-22 05:43:13][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 05:43:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9499 [2026-10-22 05:43:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0133 [2026-10-22 05:43:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9202 [2026-10-22 05:43:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1161 [2026-10-22 05:43:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4160 [2026-10-22 05:43:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9978 [2026-10-22 05:43:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0542 [2026-10-22 05:43:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8513 [2026-10-22 05:44:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9537 [2026-10-22 05:44:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5596 [2026-10-22 05:44:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6298 [2026-10-22 05:44:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1065 [2026-10-22 05:44:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0318 [2026-10-22 05:44:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9656 [2026-10-22 05:44:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4354 [2026-10-22 05:44:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3519 [2026-10-22 05:44:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0445 [2026-10-22 05:44:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2464 [2026-10-22 05:44:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7722 [2026-10-22 05:44:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7138 [2026-10-22 05:45:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2594 [2026-10-22 05:45:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.8061 [2026-10-22 05:45:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2852 [2026-10-22 05:45:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9834 [2026-10-22 05:45:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2550 [2026-10-22 05:45:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3510 [2026-10-22 05:45:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2321 [2026-10-22 05:45:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7651 [2026-10-22 05:45:31][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,583 | MFU: 1.8% [2026-10-22 05:45:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9252 [2026-10-22 05:45:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0370 [2026-10-22 05:45:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1545 [2026-10-22 05:45:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9850 [2026-10-22 05:45:37][INFO][training.pretrain_engine] [Ph2|Step 576] loss=4.9899 | ppl=146.93 | aux=0.02505 | lr=2.97e-04 | gnorm=0.645 | tok/s=29,281 | MFU=7.0% | step=143241ms | tokens=1.406B | mem=4.2/32.7GB | cost=$47.25/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 14:43 (129.0h) | Phase3: SKIPPED | SFT: Oct 27 19:27 (133.7h) | DPO (Final): Oct 27 20:01 (134.3h) [2026-10-22 05:45:40][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step576 | Tokens: 1.406B | Spent: $47.25 | Remaining: $952.75 [2026-10-22 05:45:40][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step576 [2026-10-22 05:45:40][INFO][training.pretrain_engine] Starting global step 577 (accumulating 32 micro-batches)... [2026-10-22 05:45:40][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 05:45:40][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-22 05:45:40][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-22 05:45:40][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-22 05:45:40][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step576 to Phase-Technologies/forge-500m-pretrain (path: phase2_step576)... [2026-10-22 05:45:40][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-22 05:45:50][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-22 05:45:50][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step576 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-22 05:45:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6335 [2026-10-22 05:45:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9196 [2026-10-22 05:45:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4898 [2026-10-22 05:45:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7066 [2026-10-22 05:46:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2238 [2026-10-22 05:46:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1947 [2026-10-22 05:46:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9439 [2026-10-22 05:46:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6810 [2026-10-22 05:46:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2602 [2026-10-22 05:46:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9622 [2026-10-22 05:46:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8206 [2026-10-22 05:46:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9562 [2026-10-22 05:46:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8201 [2026-10-22 05:46:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1146 [2026-10-22 05:46:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4965 [2026-10-22 05:46:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5493 [2026-10-22 05:47:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6792 [2026-10-22 05:47:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6700 [2026-10-22 05:47:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9658 [2026-10-22 05:47:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2711 [2026-10-22 05:47:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7102 [2026-10-22 05:47:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6780 [2026-10-22 05:47:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8762 [2026-10-22 05:47:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2608 [2026-10-22 05:47:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8879 [2026-10-22 05:47:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3193 [2026-10-22 05:47:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1801 [2026-10-22 05:47:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7831 [2026-10-22 05:47:58][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,582 | MFU: 1.8% [2026-10-22 05:48:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0830 [2026-10-22 05:48:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3865 [2026-10-22 05:48:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8155 [2026-10-22 05:48:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2848 [2026-10-22 05:48:03][INFO][training.pretrain_engine] [Ph2|Step 577] loss=4.9088 | ppl=135.48 | aux=0.02505 | lr=2.97e-04 | gnorm=0.559 | tok/s=29,283 | MFU=7.0% | step=143235ms | tokens=1.407B | mem=4.2/32.7GB | cost=$47.33/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 14:44 (128.9h) | Phase3: SKIPPED | SFT: Oct 27 19:29 (133.7h) | DPO (Final): Oct 27 20:03 (134.3h) [2026-10-22 05:48:03][INFO][training.pretrain_engine] Starting global step 578 (accumulating 32 micro-batches)... [2026-10-22 05:48:03][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 05:48:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3129 [2026-10-22 05:48:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4962 [2026-10-22 05:48:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4199 [2026-10-22 05:48:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2711 [2026-10-22 05:48:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7133 [2026-10-22 05:48:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3218 [2026-10-22 05:48:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7893 [2026-10-22 05:48:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6324 [2026-10-22 05:48:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9490 [2026-10-22 05:48:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9292 [2026-10-22 05:48:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8376 [2026-10-22 05:48:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3640 [2026-10-22 05:49:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1696 [2026-10-22 05:49:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1223 [2026-10-22 05:49:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1076 [2026-10-22 05:49:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0794 [2026-10-22 05:49:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2274 [2026-10-22 05:49:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8601 [2026-10-22 05:49:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9639 [2026-10-22 05:49:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7086 [2026-10-22 05:49:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7770 [2026-10-22 05:49:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8376 [2026-10-22 05:49:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8072 [2026-10-22 05:49:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9518 [2026-10-22 05:50:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9935 [2026-10-22 05:50:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8707 [2026-10-22 05:50:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7145 [2026-10-22 05:50:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1462 [2026-10-22 05:50:21][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,585 | MFU: 1.8% [2026-10-22 05:50:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2324 [2026-10-22 05:50:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2140 [2026-10-22 05:50:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6429 [2026-10-22 05:50:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8485 [2026-10-22 05:50:26][INFO][training.pretrain_engine] [Ph2|Step 578] loss=4.8780 | ppl=131.37 | aux=0.02506 | lr=2.97e-04 | gnorm=0.520 | tok/s=29,282 | MFU=7.0% | step=143238ms | tokens=1.408B | mem=4.2/32.7GB | cost=$47.41/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 14:46 (128.9h) | Phase3: SKIPPED | SFT: Oct 27 19:31 (133.7h) | DPO (Final): Oct 27 20:05 (134.2h) [2026-10-22 05:50:26][INFO][training.pretrain_engine] Starting global step 579 (accumulating 32 micro-batches)... [2026-10-22 05:50:26][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 05:50:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.0768 [2026-10-22 05:50:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8759 [2026-10-22 05:50:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6695 [2026-10-22 05:50:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7022 [2026-10-22 05:51:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6573 [2026-10-22 05:51:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6597 [2026-10-22 05:51:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2582 [2026-10-22 05:51:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1064 [2026-10-22 05:51:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0393 [2026-10-22 05:51:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2610 [2026-10-22 05:51:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6323 [2026-10-22 05:51:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7282 [2026-10-22 05:51:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4433 [2026-10-22 05:51:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8874 [2026-10-22 05:51:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1903 [2026-10-22 05:51:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9276 [2026-10-22 05:51:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6491 [2026-10-22 05:51:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8564 [2026-10-22 05:51:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9360 [2026-10-22 05:51:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3797 [2026-10-22 05:52:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4271 [2026-10-22 05:52:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1140 [2026-10-22 05:52:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8435 [2026-10-22 05:52:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8466 [2026-10-22 05:52:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3200 [2026-10-22 05:52:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1233 [2026-10-22 05:52:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3767 [2026-10-22 05:52:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0394 [2026-10-22 05:52:44][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,578 | MFU: 1.8% [2026-10-22 05:52:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1976 [2026-10-22 05:52:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6343 [2026-10-22 05:52:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2094 [2026-10-22 05:52:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9574 [2026-10-22 05:52:50][INFO][training.pretrain_engine] [Ph2|Step 579] loss=4.9598 | ppl=142.56 | aux=0.02507 | lr=2.97e-04 | gnorm=0.602 | tok/s=29,283 | MFU=7.0% | step=143235ms | tokens=1.409B | mem=4.2/32.7GB | cost=$47.49/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 14:48 (128.9h) | Phase3: SKIPPED | SFT: Oct 27 19:32 (133.7h) | DPO (Final): Oct 27 20:06 (134.2h) [2026-10-22 05:52:50][INFO][training.pretrain_engine] Starting global step 580 (accumulating 32 micro-batches)... [2026-10-22 05:52:50][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 05:53:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3006 [2026-10-22 05:53:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7997 [2026-10-22 05:53:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.1425 [2026-10-22 05:53:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3675 [2026-10-22 05:53:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0267 [2026-10-22 05:53:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7047 [2026-10-22 05:53:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1530 [2026-10-22 05:53:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8461 [2026-10-22 05:53:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7981 [2026-10-22 05:53:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7139 [2026-10-22 05:53:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7765 [2026-10-22 05:53:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5423 [2026-10-22 05:54:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4894 [2026-10-22 05:54:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3431 [2026-10-22 05:54:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2494 [2026-10-22 05:54:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1827 [2026-10-22 05:54:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9661 [2026-10-22 05:54:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0219 [2026-10-22 05:54:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3485 [2026-10-22 05:54:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7165 [2026-10-22 05:54:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8368 [2026-10-22 05:54:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4374 [2026-10-22 05:54:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8592 [2026-10-22 05:54:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6116 [2026-10-22 05:54:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0296 [2026-10-22 05:54:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7401 [2026-10-22 05:54:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.1343 [2026-10-22 05:54:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5951 [2026-10-22 05:55:08][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,567 | MFU: 1.8% [2026-10-22 05:55:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8418 [2026-10-22 05:55:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4555 [2026-10-22 05:55:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.0862 [2026-10-22 05:55:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2930 [2026-10-22 05:55:13][INFO][training.pretrain_engine] [Ph2|Step 580] loss=4.8291 | ppl=125.10 | aux=0.02506 | lr=2.97e-04 | gnorm=0.598 | tok/s=29,285 | MFU=7.0% | step=143226ms | tokens=1.410B | mem=4.2/32.7GB | cost=$47.58/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 14:49 (128.9h) | Phase3: SKIPPED | SFT: Oct 27 19:34 (133.6h) | DPO (Final): Oct 27 20:08 (134.2h) [2026-10-22 05:55:13][INFO][training.pretrain_engine] Starting global step 581 (accumulating 32 micro-batches)... [2026-10-22 05:55:13][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 05:55:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5876 [2026-10-22 05:55:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8828 [2026-10-22 05:55:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8466 [2026-10-22 05:55:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7780 [2026-10-22 05:55:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1923 [2026-10-22 05:55:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1567 [2026-10-22 05:55:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0613 [2026-10-22 05:55:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2205 [2026-10-22 05:56:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8080 [2026-10-22 05:56:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9200 [2026-10-22 05:56:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8485 [2026-10-22 05:56:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8245 [2026-10-22 05:56:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8645 [2026-10-22 05:56:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1347 [2026-10-22 05:56:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0517 [2026-10-22 05:56:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9068 [2026-10-22 05:56:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8242 [2026-10-22 05:56:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8047 [2026-10-22 05:56:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7461 [2026-10-22 05:56:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9294 [2026-10-22 05:57:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1361 [2026-10-22 05:57:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5840 [2026-10-22 05:57:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7623 [2026-10-22 05:57:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6793 [2026-10-22 05:57:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1967 [2026-10-22 05:57:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4997 [2026-10-22 05:57:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3759 [2026-10-22 05:57:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5765 [2026-10-22 05:57:31][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,563 | MFU: 1.8% [2026-10-22 05:57:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4029 [2026-10-22 05:57:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5280 [2026-10-22 05:57:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8994 [2026-10-22 05:57:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.0404 [2026-10-22 05:57:36][INFO][training.pretrain_engine] [Ph2|Step 581] loss=4.9091 | ppl=135.52 | aux=0.02506 | lr=2.97e-04 | gnorm=0.586 | tok/s=29,285 | MFU=7.0% | step=143226ms | tokens=1.411B | mem=4.2/32.7GB | cost=$47.66/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 14:51 (128.9h) | Phase3: SKIPPED | SFT: Oct 27 19:35 (133.6h) | DPO (Final): Oct 27 20:09 (134.2h) [2026-10-22 05:57:36][INFO][training.pretrain_engine] Starting global step 582 (accumulating 32 micro-batches)... [2026-10-22 05:57:36][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 05:57:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5020 [2026-10-22 05:57:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9046 [2026-10-22 05:57:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5853 [2026-10-22 05:57:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3660 [2026-10-22 05:58:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5408 [2026-10-22 05:58:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7905 [2026-10-22 05:58:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4826 [2026-10-22 05:58:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0539 [2026-10-22 05:58:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0250 [2026-10-22 05:58:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1416 [2026-10-22 05:58:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1449 [2026-10-22 05:58:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7050 [2026-10-22 05:58:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4445 [2026-10-22 05:58:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2941 [2026-10-22 05:58:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8889 [2026-10-22 05:58:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1823 [2026-10-22 05:59:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1246 [2026-10-22 05:59:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7383 [2026-10-22 05:59:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9125 [2026-10-22 05:59:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0382 [2026-10-22 05:59:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5795 [2026-10-22 05:59:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5075 [2026-10-22 05:59:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8776 [2026-10-22 05:59:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2420 [2026-10-22 05:59:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0260 [2026-10-22 05:59:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7777 [2026-10-22 05:59:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1424 [2026-10-22 05:59:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3621 [2026-10-22 05:59:55][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,552 | MFU: 1.8% [2026-10-22 06:00:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4153 [2026-10-22 06:00:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9879 [2026-10-22 06:00:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0353 [2026-10-22 06:00:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8269 [2026-10-22 06:00:01][INFO][training.pretrain_engine] [Ph2|Step 582] loss=4.7636 | ppl=117.17 | aux=0.02505 | lr=2.97e-04 | gnorm=0.590 | tok/s=29,281 | MFU=7.0% | step=143243ms | tokens=1.412B | mem=4.2/32.7GB | cost=$47.74/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 14:54 (128.9h) | Phase3: SKIPPED | SFT: Oct 27 19:38 (133.6h) | DPO (Final): Oct 27 20:12 (134.2h) [2026-10-22 06:00:01][INFO][training.pretrain_engine] Starting global step 583 (accumulating 32 micro-batches)... [2026-10-22 06:00:01][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 06:00:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1930 [2026-10-22 06:00:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8999 [2026-10-22 06:00:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8720 [2026-10-22 06:00:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0974 [2026-10-22 06:00:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7822 [2026-10-22 06:00:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8131 [2026-10-22 06:00:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1197 [2026-10-22 06:00:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0374 [2026-10-22 06:00:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3799 [2026-10-22 06:00:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1535 [2026-10-22 06:00:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5737 [2026-10-22 06:00:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9003 [2026-10-22 06:01:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6713 [2026-10-22 06:01:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9683 [2026-10-22 06:01:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1122 [2026-10-22 06:01:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8739 [2026-10-22 06:01:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9809 [2026-10-22 06:01:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1240 [2026-10-22 06:01:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7275 [2026-10-22 06:01:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9309 [2026-10-22 06:01:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=3.6619 [2026-10-22 06:01:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.1064 [2026-10-22 06:01:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3807 [2026-10-22 06:01:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=3.5944 [2026-10-22 06:02:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0975 [2026-10-22 06:02:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9245 [2026-10-22 06:02:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9220 [2026-10-22 06:02:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5082 [2026-10-22 06:02:19][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,575 | MFU: 1.8% [2026-10-22 06:02:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0662 [2026-10-22 06:02:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8194 [2026-10-22 06:02:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9902 [2026-10-22 06:02:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2898 [2026-10-22 06:02:25][INFO][training.pretrain_engine] [Ph2|Step 583] loss=4.8090 | ppl=122.61 | aux=0.02505 | lr=2.97e-04 | gnorm=0.645 | tok/s=29,280 | MFU=7.0% | step=143247ms | tokens=1.413B | mem=4.2/32.7GB | cost=$47.82/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 14:56 (128.9h) | Phase3: SKIPPED | SFT: Oct 27 19:40 (133.6h) | DPO (Final): Oct 27 20:14 (134.2h) [2026-10-22 06:02:25][INFO][training.pretrain_engine] Starting global step 584 (accumulating 32 micro-batches)... [2026-10-22 06:02:25][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 06:02:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6474 [2026-10-22 06:02:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7735 [2026-10-22 06:02:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2804 [2026-10-22 06:02:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6058 [2026-10-22 06:03:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2932 [2026-10-22 06:03:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2969 [2026-10-22 06:03:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0630 [2026-10-22 06:03:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2577 [2026-10-22 06:03:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.1959 [2026-10-22 06:03:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8707 [2026-10-22 06:03:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2973 [2026-10-22 06:03:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3333 [2026-10-22 06:03:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3253 [2026-10-22 06:03:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3767 [2026-10-22 06:03:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2967 [2026-10-22 06:03:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.1522 [2026-10-22 06:03:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3350 [2026-10-22 06:03:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8000 [2026-10-22 06:03:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0967 [2026-10-22 06:03:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8717 [2026-10-22 06:04:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6979 [2026-10-22 06:04:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3126 [2026-10-22 06:04:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6379 [2026-10-22 06:04:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5273 [2026-10-22 06:04:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6623 [2026-10-22 06:04:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2760 [2026-10-22 06:04:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7119 [2026-10-22 06:04:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8854 [2026-10-22 06:04:43][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,594 | MFU: 1.8% [2026-10-22 06:04:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5896 [2026-10-22 06:04:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1329 [2026-10-22 06:04:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1116 [2026-10-22 06:04:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1950 [2026-10-22 06:04:48][INFO][training.pretrain_engine] [Ph2|Step 584] loss=4.7692 | ppl=117.82 | aux=0.02506 | lr=2.97e-04 | gnorm=0.695 | tok/s=29,280 | MFU=7.0% | step=143248ms | tokens=1.415B | mem=4.2/32.7GB | cost=$47.90/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 14:57 (128.9h) | Phase3: SKIPPED | SFT: Oct 27 19:42 (133.6h) | DPO (Final): Oct 27 20:16 (134.2h) [2026-10-22 06:04:48][INFO][training.pretrain_engine] Starting global step 585 (accumulating 32 micro-batches)... [2026-10-22 06:04:48][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 06:05:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6559 [2026-10-22 06:05:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.6314 [2026-10-22 06:05:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.9804 [2026-10-22 06:05:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.9981 [2026-10-22 06:05:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9295 [2026-10-22 06:05:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3147 [2026-10-22 06:05:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6056 [2026-10-22 06:05:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0247 [2026-10-22 06:05:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1105 [2026-10-22 06:05:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0577 [2026-10-22 06:05:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2828 [2026-10-22 06:05:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5627 [2026-10-22 06:05:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3165 [2026-10-22 06:05:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4503 [2026-10-22 06:05:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0456 [2026-10-22 06:06:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0121 [2026-10-22 06:06:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1668 [2026-10-22 06:06:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2678 [2026-10-22 06:06:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6838 [2026-10-22 06:06:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4688 [2026-10-22 06:06:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8821 [2026-10-22 06:06:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3020 [2026-10-22 06:06:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9799 [2026-10-22 06:06:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1995 [2026-10-22 06:06:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8236 [2026-10-22 06:06:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3517 [2026-10-22 06:06:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0178 [2026-10-22 06:06:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8052 [2026-10-22 06:07:06][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,593 | MFU: 1.8% [2026-10-22 06:07:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2902 [2026-10-22 06:07:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1079 [2026-10-22 06:07:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0356 [2026-10-22 06:07:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1619 [2026-10-22 06:07:12][INFO][training.pretrain_engine] [Ph2|Step 585] loss=4.8570 | ppl=128.64 | aux=0.02507 | lr=2.97e-04 | gnorm=0.762 | tok/s=29,278 | MFU=7.0% | step=143256ms | tokens=1.416B | mem=4.2/32.7GB | cost=$47.99/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 15:00 (128.9h) | Phase3: SKIPPED | SFT: Oct 27 19:44 (133.6h) | DPO (Final): Oct 27 20:18 (134.2h) [2026-10-22 06:07:16][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step585 | Tokens: 1.416B | Spent: $47.99 | Remaining: $952.01 [2026-10-22 06:07:16][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step585 [2026-10-22 06:07:16][INFO][training.pretrain_engine] Starting global step 586 (accumulating 32 micro-batches)... [2026-10-22 06:07:16][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 06:07:16][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-22 06:07:16][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-22 06:07:16][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-22 06:07:16][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step585 to Phase-Technologies/forge-500m-pretrain (path: phase2_step585)... [2026-10-22 06:07:16][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-22 06:07:25][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-22 06:07:25][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step585 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-22 06:07:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0034 [2026-10-22 06:07:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8042 [2026-10-22 06:07:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1620 [2026-10-22 06:07:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2701 [2026-10-22 06:07:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9402 [2026-10-22 06:07:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3970 [2026-10-22 06:07:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1175 [2026-10-22 06:07:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5948 [2026-10-22 06:08:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9960 [2026-10-22 06:08:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9456 [2026-10-22 06:08:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9065 [2026-10-22 06:08:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8037 [2026-10-22 06:08:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0575 [2026-10-22 06:08:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8698 [2026-10-22 06:08:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0653 [2026-10-22 06:08:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1160 [2026-10-22 06:08:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3240 [2026-10-22 06:08:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0594 [2026-10-22 06:08:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8707 [2026-10-22 06:08:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6446 [2026-10-22 06:08:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9632 [2026-10-22 06:09:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8458 [2026-10-22 06:09:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2345 [2026-10-22 06:09:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0359 [2026-10-22 06:09:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7629 [2026-10-22 06:09:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6264 [2026-10-22 06:09:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7648 [2026-10-22 06:09:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3222 [2026-10-22 06:09:34][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,592 | MFU: 1.8% [2026-10-22 06:09:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9031 [2026-10-22 06:09:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8436 [2026-10-22 06:09:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5407 [2026-10-22 06:09:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9010 [2026-10-22 06:09:38][INFO][training.pretrain_engine] [Ph2|Step 586] loss=5.0073 | ppl=149.50 | aux=0.02505 | lr=2.97e-04 | gnorm=0.766 | tok/s=29,280 | MFU=7.0% | step=143249ms | tokens=1.417B | mem=4.2/32.7GB | cost=$48.07/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 15:01 (128.9h) | Phase3: SKIPPED | SFT: Oct 27 19:46 (133.6h) | DPO (Final): Oct 27 20:20 (134.2h) [2026-10-22 06:09:38][INFO][training.pretrain_engine] Starting global step 587 (accumulating 32 micro-batches)... [2026-10-22 06:09:38][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 06:09:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4792 [2026-10-22 06:09:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9039 [2026-10-22 06:09:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1359 [2026-10-22 06:09:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7240 [2026-10-22 06:10:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8001 [2026-10-22 06:10:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7375 [2026-10-22 06:10:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4756 [2026-10-22 06:10:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8958 [2026-10-22 06:10:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2878 [2026-10-22 06:10:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9876 [2026-10-22 06:10:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1802 [2026-10-22 06:10:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0622 [2026-10-22 06:10:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9328 [2026-10-22 06:10:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1849 [2026-10-22 06:10:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0465 [2026-10-22 06:10:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2658 [2026-10-22 06:11:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7596 [2026-10-22 06:11:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8724 [2026-10-22 06:11:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9919 [2026-10-22 06:11:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6889 [2026-10-22 06:11:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9551 [2026-10-22 06:11:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7307 [2026-10-22 06:11:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2009 [2026-10-22 06:11:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0085 [2026-10-22 06:11:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9316 [2026-10-22 06:11:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2439 [2026-10-22 06:11:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9490 [2026-10-22 06:11:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6696 [2026-10-22 06:11:57][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,583 | MFU: 1.8% [2026-10-22 06:12:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2664 [2026-10-22 06:12:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5255 [2026-10-22 06:12:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7762 [2026-10-22 06:12:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2451 [2026-10-22 06:12:02][INFO][training.pretrain_engine] [Ph2|Step 587] loss=4.8268 | ppl=124.81 | aux=0.02506 | lr=2.97e-04 | gnorm=0.754 | tok/s=29,277 | MFU=7.0% | step=143262ms | tokens=1.418B | mem=4.2/32.7GB | cost=$48.15/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 15:04 (128.9h) | Phase3: SKIPPED | SFT: Oct 27 19:48 (133.6h) | DPO (Final): Oct 27 20:22 (134.2h) [2026-10-22 06:12:02][INFO][training.pretrain_engine] Starting global step 588 (accumulating 32 micro-batches)... [2026-10-22 06:12:02][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 06:12:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8414 [2026-10-22 06:12:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8137 [2026-10-22 06:12:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3479 [2026-10-22 06:12:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0163 [2026-10-22 06:12:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6019 [2026-10-22 06:12:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8519 [2026-10-22 06:12:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7951 [2026-10-22 06:12:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8713 [2026-10-22 06:12:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4797 [2026-10-22 06:12:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7914 [2026-10-22 06:12:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5532 [2026-10-22 06:12:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6866 [2026-10-22 06:13:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9028 [2026-10-22 06:13:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.1486 [2026-10-22 06:13:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7479 [2026-10-22 06:13:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6292 [2026-10-22 06:13:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5079 [2026-10-22 06:13:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8351 [2026-10-22 06:13:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8564 [2026-10-22 06:13:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8578 [2026-10-22 06:13:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9840 [2026-10-22 06:13:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9770 [2026-10-22 06:13:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1434 [2026-10-22 06:13:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8199 [2026-10-22 06:14:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8837 [2026-10-22 06:14:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0285 [2026-10-22 06:14:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8938 [2026-10-22 06:14:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7775 [2026-10-22 06:14:21][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,562 | MFU: 1.8% [2026-10-22 06:14:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4298 [2026-10-22 06:14:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8105 [2026-10-22 06:14:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3290 [2026-10-22 06:14:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6093 [2026-10-22 06:14:26][INFO][training.pretrain_engine] [Ph2|Step 588] loss=4.8480 | ppl=127.48 | aux=0.02504 | lr=2.97e-04 | gnorm=0.852 | tok/s=29,275 | MFU=7.0% | step=143272ms | tokens=1.419B | mem=4.2/32.7GB | cost=$48.24/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 15:06 (128.9h) | Phase3: SKIPPED | SFT: Oct 27 19:51 (133.6h) | DPO (Final): Oct 27 20:25 (134.2h) [2026-10-22 06:14:26][INFO][training.pretrain_engine] Starting global step 589 (accumulating 32 micro-batches)... [2026-10-22 06:14:26][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 06:14:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5903 [2026-10-22 06:14:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8815 [2026-10-22 06:14:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.1543 [2026-10-22 06:14:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9728 [2026-10-22 06:15:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.7894 [2026-10-22 06:15:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3351 [2026-10-22 06:15:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1335 [2026-10-22 06:15:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1286 [2026-10-22 06:15:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3529 [2026-10-22 06:15:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6822 [2026-10-22 06:15:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8346 [2026-10-22 06:15:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1421 [2026-10-22 06:15:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9367 [2026-10-22 06:15:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6612 [2026-10-22 06:15:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7811 [2026-10-22 06:15:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9159 [2026-10-22 06:15:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4493 [2026-10-22 06:15:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4635 [2026-10-22 06:15:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5562 [2026-10-22 06:15:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5707 [2026-10-22 06:16:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3183 [2026-10-22 06:16:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8266 [2026-10-22 06:16:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8347 [2026-10-22 06:16:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6729 [2026-10-22 06:16:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0148 [2026-10-22 06:16:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0728 [2026-10-22 06:16:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0643 [2026-10-22 06:16:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1526 [2026-10-22 06:16:45][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,582 | MFU: 1.8% [2026-10-22 06:16:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1724 [2026-10-22 06:16:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0522 [2026-10-22 06:16:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9971 [2026-10-22 06:16:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0610 [2026-10-22 06:16:50][INFO][training.pretrain_engine] [Ph2|Step 589] loss=4.8302 | ppl=125.23 | aux=0.02506 | lr=2.97e-04 | gnorm=0.910 | tok/s=29,274 | MFU=7.0% | step=143278ms | tokens=1.420B | mem=4.2/32.7GB | cost=$48.32/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 15:08 (128.9h) | Phase3: SKIPPED | SFT: Oct 27 19:53 (133.6h) | DPO (Final): Oct 27 20:27 (134.2h) [2026-10-22 06:16:50][INFO][training.pretrain_engine] Starting global step 590 (accumulating 32 micro-batches)... [2026-10-22 06:16:50][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 06:17:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0693 [2026-10-22 06:17:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8308 [2026-10-22 06:17:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1255 [2026-10-22 06:17:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0140 [2026-10-22 06:17:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1134 [2026-10-22 06:17:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0460 [2026-10-22 06:17:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5075 [2026-10-22 06:17:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9062 [2026-10-22 06:17:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6404 [2026-10-22 06:17:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5706 [2026-10-22 06:17:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0226 [2026-10-22 06:17:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9253 [2026-10-22 06:18:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4696 [2026-10-22 06:18:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1300 [2026-10-22 06:18:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5089 [2026-10-22 06:18:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9870 [2026-10-22 06:18:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0829 [2026-10-22 06:18:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8727 [2026-10-22 06:18:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0691 [2026-10-22 06:18:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9140 [2026-10-22 06:18:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1840 [2026-10-22 06:18:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0212 [2026-10-22 06:18:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1981 [2026-10-22 06:18:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9400 [2026-10-22 06:18:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6646 [2026-10-22 06:18:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9717 [2026-10-22 06:18:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7909 [2026-10-22 06:18:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6280 [2026-10-22 06:19:09][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,562 | MFU: 1.8% [2026-10-22 06:19:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2663 [2026-10-22 06:19:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5668 [2026-10-22 06:19:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3059 [2026-10-22 06:19:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0674 [2026-10-22 06:19:14][INFO][training.pretrain_engine] [Ph2|Step 590] loss=4.8997 | ppl=134.25 | aux=0.02505 | lr=2.97e-04 | gnorm=0.809 | tok/s=29,272 | MFU=7.0% | step=143285ms | tokens=1.421B | mem=4.2/32.7GB | cost=$48.40/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 15:10 (128.9h) | Phase3: SKIPPED | SFT: Oct 27 19:55 (133.6h) | DPO (Final): Oct 27 20:29 (134.2h) [2026-10-22 06:19:14][INFO][training.pretrain_engine] Starting global step 591 (accumulating 32 micro-batches)... [2026-10-22 06:19:14][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 06:19:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9031 [2026-10-22 06:19:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9937 [2026-10-22 06:19:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7341 [2026-10-22 06:19:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0067 [2026-10-22 06:19:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8084 [2026-10-22 06:19:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9605 [2026-10-22 06:19:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7655 [2026-10-22 06:19:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8085 [2026-10-22 06:20:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4924 [2026-10-22 06:20:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3138 [2026-10-22 06:20:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4926 [2026-10-22 06:20:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0000 [2026-10-22 06:20:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5498 [2026-10-22 06:20:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1666 [2026-10-22 06:20:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6580 [2026-10-22 06:20:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0145 [2026-10-22 06:20:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2348 [2026-10-22 06:20:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7346 [2026-10-22 06:20:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1556 [2026-10-22 06:20:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1409 [2026-10-22 06:21:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8956 [2026-10-22 06:21:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9423 [2026-10-22 06:21:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7482 [2026-10-22 06:21:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2005 [2026-10-22 06:21:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5516 [2026-10-22 06:21:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4225 [2026-10-22 06:21:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6914 [2026-10-22 06:21:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5460 [2026-10-22 06:21:32][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,567 | MFU: 1.8% [2026-10-22 06:21:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6625 [2026-10-22 06:21:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1417 [2026-10-22 06:21:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1305 [2026-10-22 06:21:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0010 [2026-10-22 06:21:37][INFO][training.pretrain_engine] [Ph2|Step 591] loss=4.8411 | ppl=126.61 | aux=0.02506 | lr=2.97e-04 | gnorm=0.844 | tok/s=29,271 | MFU=7.0% | step=143292ms | tokens=1.422B | mem=4.2/32.7GB | cost=$48.48/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 15:12 (128.9h) | Phase3: SKIPPED | SFT: Oct 27 19:57 (133.6h) | DPO (Final): Oct 27 20:31 (134.2h) [2026-10-22 06:21:37][INFO][training.pretrain_engine] Starting global step 592 (accumulating 32 micro-batches)... [2026-10-22 06:21:37][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 06:21:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9870 [2026-10-22 06:21:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1825 [2026-10-22 06:21:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1808 [2026-10-22 06:21:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0253 [2026-10-22 06:22:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2327 [2026-10-22 06:22:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6086 [2026-10-22 06:22:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.6944 [2026-10-22 06:22:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3255 [2026-10-22 06:22:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9295 [2026-10-22 06:22:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3475 [2026-10-22 06:22:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7964 [2026-10-22 06:22:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0306 [2026-10-22 06:22:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4523 [2026-10-22 06:22:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6870 [2026-10-22 06:22:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7206 [2026-10-22 06:22:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3556 [2026-10-22 06:23:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0696 [2026-10-22 06:23:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5590 [2026-10-22 06:23:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7072 [2026-10-22 06:23:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6299 [2026-10-22 06:23:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9250 [2026-10-22 06:23:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5889 [2026-10-22 06:23:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6657 [2026-10-22 06:23:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3592 [2026-10-22 06:23:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1334 [2026-10-22 06:23:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9153 [2026-10-22 06:23:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9715 [2026-10-22 06:23:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2872 [2026-10-22 06:23:56][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,570 | MFU: 1.8% [2026-10-22 06:24:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9953 [2026-10-22 06:24:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9309 [2026-10-22 06:24:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2681 [2026-10-22 06:24:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1095 [2026-10-22 06:24:01][INFO][training.pretrain_engine] [Ph2|Step 592] loss=4.8913 | ppl=133.12 | aux=0.02506 | lr=2.97e-04 | gnorm=0.707 | tok/s=29,270 | MFU=7.0% | step=143295ms | tokens=1.423B | mem=4.2/32.7GB | cost=$48.56/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 15:14 (128.8h) | Phase3: SKIPPED | SFT: Oct 27 19:59 (133.6h) | DPO (Final): Oct 27 20:33 (134.2h) [2026-10-22 06:24:01][INFO][training.pretrain_engine] Starting global step 593 (accumulating 32 micro-batches)... [2026-10-22 06:24:01][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 06:24:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1328 [2026-10-22 06:24:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8200 [2026-10-22 06:24:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6569 [2026-10-22 06:24:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9121 [2026-10-22 06:24:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8220 [2026-10-22 06:24:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6945 [2026-10-22 06:24:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7995 [2026-10-22 06:24:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0848 [2026-10-22 06:24:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8290 [2026-10-22 06:24:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6942 [2026-10-22 06:24:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5108 [2026-10-22 06:24:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7542 [2026-10-22 06:25:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7150 [2026-10-22 06:25:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9415 [2026-10-22 06:25:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0963 [2026-10-22 06:25:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1256 [2026-10-22 06:25:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0450 [2026-10-22 06:25:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1359 [2026-10-22 06:25:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2838 [2026-10-22 06:25:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0269 [2026-10-22 06:25:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3124 [2026-10-22 06:25:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0564 [2026-10-22 06:25:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9050 [2026-10-22 06:25:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1222 [2026-10-22 06:26:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1220 [2026-10-22 06:26:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0055 [2026-10-22 06:26:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8472 [2026-10-22 06:26:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0183 [2026-10-22 06:26:19][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,587 | MFU: 1.8% [2026-10-22 06:26:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9272 [2026-10-22 06:26:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9574 [2026-10-22 06:26:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8645 [2026-10-22 06:26:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9965 [2026-10-22 06:26:25][INFO][training.pretrain_engine] [Ph2|Step 593] loss=4.9950 | ppl=147.67 | aux=0.02506 | lr=2.97e-04 | gnorm=0.613 | tok/s=29,269 | MFU=7.0% | step=143301ms | tokens=1.424B | mem=4.2/32.7GB | cost=$48.65/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 15:16 (128.8h) | Phase3: SKIPPED | SFT: Oct 27 20:01 (133.6h) | DPO (Final): Oct 27 20:35 (134.2h) [2026-10-22 06:26:25][INFO][training.pretrain_engine] Starting global step 594 (accumulating 32 micro-batches)... [2026-10-22 06:26:25][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 06:26:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8446 [2026-10-22 06:26:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8559 [2026-10-22 06:26:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6848 [2026-10-22 06:26:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8858 [2026-10-22 06:27:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4824 [2026-10-22 06:27:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4276 [2026-10-22 06:27:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.0646 [2026-10-22 06:27:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5204 [2026-10-22 06:27:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7834 [2026-10-22 06:27:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6971 [2026-10-22 06:27:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5679 [2026-10-22 06:27:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7424 [2026-10-22 06:27:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9848 [2026-10-22 06:27:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8599 [2026-10-22 06:27:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6670 [2026-10-22 06:27:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8300 [2026-10-22 06:27:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=3.8410 [2026-10-22 06:27:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4117 [2026-10-22 06:27:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5096 [2026-10-22 06:27:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3745 [2026-10-22 06:28:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5728 [2026-10-22 06:28:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4792 [2026-10-22 06:28:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7793 [2026-10-22 06:28:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6297 [2026-10-22 06:28:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0963 [2026-10-22 06:28:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0442 [2026-10-22 06:28:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5900 [2026-10-22 06:28:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5483 [2026-10-22 06:28:43][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,575 | MFU: 1.8% [2026-10-22 06:28:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9360 [2026-10-22 06:28:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8461 [2026-10-22 06:28:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6118 [2026-10-22 06:28:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0441 [2026-10-22 06:28:48][INFO][training.pretrain_engine] [Ph2|Step 594] loss=4.8227 | ppl=124.31 | aux=0.02506 | lr=2.97e-04 | gnorm=0.566 | tok/s=29,269 | MFU=7.0% | step=143302ms | tokens=1.425B | mem=4.2/32.7GB | cost=$48.73/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 15:18 (128.8h) | Phase3: SKIPPED | SFT: Oct 27 20:03 (133.6h) | DPO (Final): Oct 27 20:37 (134.1h) [2026-10-22 06:28:51][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step594 | Tokens: 1.425B | Spent: $48.73 | Remaining: $951.27 [2026-10-22 06:28:51][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step594 [2026-10-22 06:28:52][INFO][training.pretrain_engine] Starting global step 595 (accumulating 32 micro-batches)... [2026-10-22 06:28:52][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 06:28:52][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-22 06:28:52][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-22 06:28:52][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-22 06:28:52][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step594 to Phase-Technologies/forge-500m-pretrain (path: phase2_step594)... [2026-10-22 06:28:52][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-22 06:29:01][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-22 06:29:01][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step594 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-22 06:29:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6691 [2026-10-22 06:29:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7599 [2026-10-22 06:29:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8930 [2026-10-22 06:29:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7460 [2026-10-22 06:29:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1665 [2026-10-22 06:29:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9472 [2026-10-22 06:29:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2367 [2026-10-22 06:29:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0124 [2026-10-22 06:29:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=3.6785 [2026-10-22 06:29:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4267 [2026-10-22 06:29:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4939 [2026-10-22 06:29:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3572 [2026-10-22 06:29:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6270 [2026-10-22 06:30:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4723 [2026-10-22 06:30:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0925 [2026-10-22 06:30:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6341 [2026-10-22 06:30:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.1838 [2026-10-22 06:30:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=3.8703 [2026-10-22 06:30:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.1972 [2026-10-22 06:30:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4192 [2026-10-22 06:30:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0298 [2026-10-22 06:30:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0840 [2026-10-22 06:30:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9576 [2026-10-22 06:30:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1991 [2026-10-22 06:30:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0200 [2026-10-22 06:30:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6746 [2026-10-22 06:30:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3797 [2026-10-22 06:30:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9135 [2026-10-22 06:31:10][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,584 | MFU: 1.8% [2026-10-22 06:31:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8412 [2026-10-22 06:31:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3990 [2026-10-22 06:31:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9481 [2026-10-22 06:31:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5981 [2026-10-22 06:31:14][INFO][training.pretrain_engine] [Ph2|Step 595] loss=4.8876 | ppl=132.63 | aux=0.02506 | lr=2.97e-04 | gnorm=0.684 | tok/s=29,271 | MFU=7.0% | step=143295ms | tokens=1.426B | mem=4.2/32.7GB | cost=$48.81/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 15:20 (128.8h) | Phase3: SKIPPED | SFT: Oct 27 20:04 (133.6h) | DPO (Final): Oct 27 20:39 (134.1h) [2026-10-22 06:31:14][INFO][training.pretrain_engine] Starting global step 596 (accumulating 32 micro-batches)... [2026-10-22 06:31:14][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 06:31:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5192 [2026-10-22 06:31:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9989 [2026-10-22 06:31:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8455 [2026-10-22 06:31:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1686 [2026-10-22 06:31:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8301 [2026-10-22 06:31:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.0179 [2026-10-22 06:31:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7182 [2026-10-22 06:31:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3810 [2026-10-22 06:32:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1727 [2026-10-22 06:32:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0472 [2026-10-22 06:32:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8785 [2026-10-22 06:32:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9895 [2026-10-22 06:32:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3618 [2026-10-22 06:32:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7798 [2026-10-22 06:32:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3727 [2026-10-22 06:32:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1404 [2026-10-22 06:32:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9207 [2026-10-22 06:32:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4131 [2026-10-22 06:32:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5186 [2026-10-22 06:32:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7143 [2026-10-22 06:33:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7570 [2026-10-22 06:33:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6168 [2026-10-22 06:33:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0175 [2026-10-22 06:33:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7663 [2026-10-22 06:33:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1479 [2026-10-22 06:33:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8767 [2026-10-22 06:33:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7752 [2026-10-22 06:33:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8762 [2026-10-22 06:33:33][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,571 | MFU: 1.8% [2026-10-22 06:33:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7801 [2026-10-22 06:33:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1227 [2026-10-22 06:33:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8696 [2026-10-22 06:33:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6242 [2026-10-22 06:33:38][INFO][training.pretrain_engine] [Ph2|Step 596] loss=4.8780 | ppl=131.37 | aux=0.02505 | lr=2.97e-04 | gnorm=0.586 | tok/s=29,267 | MFU=7.0% | step=143311ms | tokens=1.427B | mem=4.2/32.7GB | cost=$48.89/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 15:22 (128.8h) | Phase3: SKIPPED | SFT: Oct 27 20:07 (133.6h) | DPO (Final): Oct 27 20:41 (134.1h) [2026-10-22 06:33:38][INFO][training.pretrain_engine] Starting global step 597 (accumulating 32 micro-batches)... [2026-10-22 06:33:38][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 06:33:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5997 [2026-10-22 06:33:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6491 [2026-10-22 06:33:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5332 [2026-10-22 06:33:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2973 [2026-10-22 06:34:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8414 [2026-10-22 06:34:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4963 [2026-10-22 06:34:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0355 [2026-10-22 06:34:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0506 [2026-10-22 06:34:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0922 [2026-10-22 06:34:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3542 [2026-10-22 06:34:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8451 [2026-10-22 06:34:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7998 [2026-10-22 06:34:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1905 [2026-10-22 06:34:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9265 [2026-10-22 06:34:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2442 [2026-10-22 06:34:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0460 [2026-10-22 06:35:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1899 [2026-10-22 06:35:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2665 [2026-10-22 06:35:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4912 [2026-10-22 06:35:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7302 [2026-10-22 06:35:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1683 [2026-10-22 06:35:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1856 [2026-10-22 06:35:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3474 [2026-10-22 06:35:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9256 [2026-10-22 06:35:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0188 [2026-10-22 06:35:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0418 [2026-10-22 06:35:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3847 [2026-10-22 06:35:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9617 [2026-10-22 06:35:57][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,590 | MFU: 1.8% [2026-10-22 06:36:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5039 [2026-10-22 06:36:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6556 [2026-10-22 06:36:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6369 [2026-10-22 06:36:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4966 [2026-10-22 06:36:02][INFO][training.pretrain_engine] [Ph2|Step 597] loss=4.9678 | ppl=143.71 | aux=0.02506 | lr=2.97e-04 | gnorm=0.535 | tok/s=29,267 | MFU=7.0% | step=143313ms | tokens=1.428B | mem=4.2/32.7GB | cost=$48.98/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 15:24 (128.8h) | Phase3: SKIPPED | SFT: Oct 27 20:09 (133.6h) | DPO (Final): Oct 27 20:43 (134.1h) [2026-10-22 06:36:02][INFO][training.pretrain_engine] Starting global step 598 (accumulating 32 micro-batches)... [2026-10-22 06:36:02][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 06:36:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0077 [2026-10-22 06:36:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5631 [2026-10-22 06:36:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9557 [2026-10-22 06:36:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0159 [2026-10-22 06:36:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5751 [2026-10-22 06:36:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6428 [2026-10-22 06:36:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.7859 [2026-10-22 06:36:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.1507 [2026-10-22 06:36:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0637 [2026-10-22 06:36:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6686 [2026-10-22 06:36:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1915 [2026-10-22 06:36:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9303 [2026-10-22 06:37:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1253 [2026-10-22 06:37:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8956 [2026-10-22 06:37:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6969 [2026-10-22 06:37:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6777 [2026-10-22 06:37:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6264 [2026-10-22 06:37:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6686 [2026-10-22 06:37:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7596 [2026-10-22 06:37:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.2426 [2026-10-22 06:37:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8494 [2026-10-22 06:37:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9773 [2026-10-22 06:37:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1539 [2026-10-22 06:37:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6329 [2026-10-22 06:38:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0770 [2026-10-22 06:38:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2058 [2026-10-22 06:38:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8326 [2026-10-22 06:38:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2943 [2026-10-22 06:38:20][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,587 | MFU: 1.8% [2026-10-22 06:38:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9296 [2026-10-22 06:38:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9967 [2026-10-22 06:38:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5709 [2026-10-22 06:38:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2983 [2026-10-22 06:38:25][INFO][training.pretrain_engine] [Ph2|Step 598] loss=4.7863 | ppl=119.86 | aux=0.02505 | lr=2.97e-04 | gnorm=0.508 | tok/s=29,266 | MFU=7.0% | step=143317ms | tokens=1.429B | mem=4.2/32.7GB | cost=$49.06/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 15:26 (128.8h) | Phase3: SKIPPED | SFT: Oct 27 20:11 (133.6h) | DPO (Final): Oct 27 20:45 (134.1h) [2026-10-22 06:38:25][INFO][training.pretrain_engine] Starting global step 599 (accumulating 32 micro-batches)... [2026-10-22 06:38:25][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 06:38:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3120 [2026-10-22 06:38:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6320 [2026-10-22 06:38:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5288 [2026-10-22 06:38:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2504 [2026-10-22 06:39:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2694 [2026-10-22 06:39:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2112 [2026-10-22 06:39:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5372 [2026-10-22 06:39:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2620 [2026-10-22 06:39:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7234 [2026-10-22 06:39:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5713 [2026-10-22 06:39:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2781 [2026-10-22 06:39:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0023 [2026-10-22 06:39:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2544 [2026-10-22 06:39:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6659 [2026-10-22 06:39:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4634 [2026-10-22 06:39:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6136 [2026-10-22 06:39:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2947 [2026-10-22 06:39:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8095 [2026-10-22 06:39:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8800 [2026-10-22 06:39:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1672 [2026-10-22 06:40:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3292 [2026-10-22 06:40:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1162 [2026-10-22 06:40:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5134 [2026-10-22 06:40:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1822 [2026-10-22 06:40:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6691 [2026-10-22 06:40:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1530 [2026-10-22 06:40:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7451 [2026-10-22 06:40:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8307 [2026-10-22 06:40:44][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,581 | MFU: 1.8% [2026-10-22 06:40:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0345 [2026-10-22 06:40:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7300 [2026-10-22 06:40:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9799 [2026-10-22 06:40:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8643 [2026-10-22 06:40:49][INFO][training.pretrain_engine] [Ph2|Step 599] loss=4.8247 | ppl=124.55 | aux=0.02507 | lr=2.97e-04 | gnorm=0.570 | tok/s=29,266 | MFU=7.0% | step=143318ms | tokens=1.430B | mem=4.2/32.7GB | cost=$49.14/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 15:28 (128.8h) | Phase3: SKIPPED | SFT: Oct 27 20:13 (133.5h) | DPO (Final): Oct 27 20:47 (134.1h) [2026-10-22 06:40:49][INFO][training.pretrain_engine] Starting global step 600 (accumulating 32 micro-batches)... [2026-10-22 06:40:49][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 06:41:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0349 [2026-10-22 06:41:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8778 [2026-10-22 06:41:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7022 [2026-10-22 06:41:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6120 [2026-10-22 06:41:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1911 [2026-10-22 06:41:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4303 [2026-10-22 06:41:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6449 [2026-10-22 06:41:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.0737 [2026-10-22 06:41:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9364 [2026-10-22 06:41:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7095 [2026-10-22 06:41:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7383 [2026-10-22 06:41:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6376 [2026-10-22 06:42:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.7052 [2026-10-22 06:42:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3884 [2026-10-22 06:42:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1493 [2026-10-22 06:42:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1768 [2026-10-22 06:42:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.0292 [2026-10-22 06:42:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1076 [2026-10-22 06:42:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0976 [2026-10-22 06:42:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8865 [2026-10-22 06:42:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6061 [2026-10-22 06:42:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3215 [2026-10-22 06:42:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9779 [2026-10-22 06:42:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6884 [2026-10-22 06:42:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=6.0353 [2026-10-22 06:42:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9536 [2026-10-22 06:42:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4498 [2026-10-22 06:42:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0200 [2026-10-22 06:43:07][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,592 | MFU: 1.8% [2026-10-22 06:43:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9886 [2026-10-22 06:43:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3294 [2026-10-22 06:43:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9019 [2026-10-22 06:43:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8776 [2026-10-22 06:43:12][INFO][training.pretrain_engine] [Ph2|Step 600] loss=4.8609 | ppl=129.14 | aux=0.02505 | lr=2.97e-04 | gnorm=0.484 | tok/s=29,265 | MFU=7.0% | step=143322ms | tokens=1.431B | mem=4.2/32.7GB | cost=$49.22/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 15:30 (128.8h) | Phase3: SKIPPED | SFT: Oct 27 20:15 (133.5h) | DPO (Final): Oct 27 20:49 (134.1h) [2026-10-22 06:43:12][INFO][training.pretrain_engine] Starting global step 601 (accumulating 32 micro-batches)... [2026-10-22 06:43:12][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 06:43:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8797 [2026-10-22 06:43:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9014 [2026-10-22 06:43:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2113 [2026-10-22 06:43:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3202 [2026-10-22 06:43:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7513 [2026-10-22 06:43:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7044 [2026-10-22 06:43:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5443 [2026-10-22 06:43:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.0391 [2026-10-22 06:44:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5449 [2026-10-22 06:44:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0643 [2026-10-22 06:44:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9230 [2026-10-22 06:44:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9030 [2026-10-22 06:44:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0642 [2026-10-22 06:44:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0597 [2026-10-22 06:44:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7260 [2026-10-22 06:44:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8642 [2026-10-22 06:44:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4562 [2026-10-22 06:44:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5477 [2026-10-22 06:44:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1468 [2026-10-22 06:44:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3378 [2026-10-22 06:44:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7283 [2026-10-22 06:44:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6995 [2026-10-22 06:45:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9156 [2026-10-22 06:45:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7727 [2026-10-22 06:45:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8836 [2026-10-22 06:45:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2296 [2026-10-22 06:45:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7768 [2026-10-22 06:45:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5528 [2026-10-22 06:45:31][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,571 | MFU: 1.8% [2026-10-22 06:45:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0963 [2026-10-22 06:45:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9384 [2026-10-22 06:45:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0774 [2026-10-22 06:45:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2848 [2026-10-22 06:45:35][INFO][training.pretrain_engine] [Ph2|Step 601] loss=4.8115 | ppl=122.91 | aux=0.02506 | lr=2.97e-04 | gnorm=0.562 | tok/s=29,265 | MFU=7.0% | step=143320ms | tokens=1.432B | mem=4.2/32.7GB | cost=$49.31/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 15:32 (128.8h) | Phase3: SKIPPED | SFT: Oct 27 20:17 (133.5h) | DPO (Final): Oct 27 20:51 (134.1h) [2026-10-22 06:45:35][INFO][training.pretrain_engine] Starting global step 602 (accumulating 32 micro-batches)... [2026-10-22 06:45:35][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 06:45:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8277 [2026-10-22 06:45:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0396 [2026-10-22 06:45:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4022 [2026-10-22 06:45:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2293 [2026-10-22 06:46:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0228 [2026-10-22 06:46:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0340 [2026-10-22 06:46:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5233 [2026-10-22 06:46:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2993 [2026-10-22 06:46:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5906 [2026-10-22 06:46:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8612 [2026-10-22 06:46:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2385 [2026-10-22 06:46:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9706 [2026-10-22 06:46:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0227 [2026-10-22 06:46:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9130 [2026-10-22 06:46:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0647 [2026-10-22 06:46:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9239 [2026-10-22 06:47:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0020 [2026-10-22 06:47:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6407 [2026-10-22 06:47:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5453 [2026-10-22 06:47:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9577 [2026-10-22 06:47:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6895 [2026-10-22 06:47:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.1376 [2026-10-22 06:47:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8495 [2026-10-22 06:47:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0776 [2026-10-22 06:47:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8388 [2026-10-22 06:47:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6660 [2026-10-22 06:47:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0535 [2026-10-22 06:47:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6687 [2026-10-22 06:47:54][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,582 | MFU: 1.8% [2026-10-22 06:47:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2845 [2026-10-22 06:47:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2436 [2026-10-22 06:47:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2106 [2026-10-22 06:47:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1198 [2026-10-22 06:47:59][INFO][training.pretrain_engine] [Ph2|Step 602] loss=4.8450 | ppl=127.10 | aux=0.02505 | lr=2.97e-04 | gnorm=0.471 | tok/s=29,265 | MFU=7.0% | step=143321ms | tokens=1.433B | mem=4.2/32.7GB | cost=$49.39/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 15:34 (128.8h) | Phase3: SKIPPED | SFT: Oct 27 20:19 (133.5h) | DPO (Final): Oct 27 20:53 (134.1h) [2026-10-22 06:47:59][INFO][training.pretrain_engine] Starting global step 603 (accumulating 32 micro-batches)... [2026-10-22 06:47:59][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 06:48:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9338 [2026-10-22 06:48:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8294 [2026-10-22 06:48:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.0435 [2026-10-22 06:48:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3249 [2026-10-22 06:48:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6056 [2026-10-22 06:48:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3876 [2026-10-22 06:48:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7636 [2026-10-22 06:48:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3175 [2026-10-22 06:48:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8174 [2026-10-22 06:48:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5717 [2026-10-22 06:48:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1947 [2026-10-22 06:48:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8528 [2026-10-22 06:49:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4675 [2026-10-22 06:49:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6079 [2026-10-22 06:49:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5540 [2026-10-22 06:49:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9296 [2026-10-22 06:49:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=3.4159 [2026-10-22 06:49:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8216 [2026-10-22 06:49:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7670 [2026-10-22 06:49:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8148 [2026-10-22 06:49:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8625 [2026-10-22 06:49:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1878 [2026-10-22 06:49:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0898 [2026-10-22 06:49:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1560 [2026-10-22 06:50:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9342 [2026-10-22 06:50:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4654 [2026-10-22 06:50:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0248 [2026-10-22 06:50:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7568 [2026-10-22 06:50:17][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,582 | MFU: 1.8% [2026-10-22 06:50:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9319 [2026-10-22 06:50:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9989 [2026-10-22 06:50:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5299 [2026-10-22 06:50:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1086 [2026-10-22 06:50:23][INFO][training.pretrain_engine] [Ph2|Step 603] loss=4.8656 | ppl=129.74 | aux=0.02506 | lr=2.97e-04 | gnorm=0.459 | tok/s=29,264 | MFU=7.0% | step=143326ms | tokens=1.434B | mem=4.2/32.7GB | cost=$49.47/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 15:36 (128.8h) | Phase3: SKIPPED | SFT: Oct 27 20:21 (133.5h) | DPO (Final): Oct 27 20:55 (134.1h) [2026-10-22 06:50:26][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step603 | Tokens: 1.434B | Spent: $49.47 | Remaining: $950.53 [2026-10-22 06:50:26][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step603 [2026-10-22 06:50:26][INFO][training.pretrain_engine] Starting global step 604 (accumulating 32 micro-batches)... [2026-10-22 06:50:26][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 06:50:26][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-22 06:50:26][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-22 06:50:26][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-22 06:50:26][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step603 to Phase-Technologies/forge-500m-pretrain (path: phase2_step603)... [2026-10-22 06:50:26][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-22 06:50:35][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-22 06:50:35][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step603 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-22 06:50:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9611 [2026-10-22 06:50:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8033 [2026-10-22 06:50:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7787 [2026-10-22 06:50:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9325 [2026-10-22 06:50:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1035 [2026-10-22 06:50:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9590 [2026-10-22 06:50:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9977 [2026-10-22 06:51:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0499 [2026-10-22 06:51:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8510 [2026-10-22 06:51:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8774 [2026-10-22 06:51:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9283 [2026-10-22 06:51:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5809 [2026-10-22 06:51:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4993 [2026-10-22 06:51:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=3.7329 [2026-10-22 06:51:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6661 [2026-10-22 06:51:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7501 [2026-10-22 06:51:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9870 [2026-10-22 06:51:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8663 [2026-10-22 06:51:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8850 [2026-10-22 06:51:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5299 [2026-10-22 06:52:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3573 [2026-10-22 06:52:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0582 [2026-10-22 06:52:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1940 [2026-10-22 06:52:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0731 [2026-10-22 06:52:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7677 [2026-10-22 06:52:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7636 [2026-10-22 06:52:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0155 [2026-10-22 06:52:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1752 [2026-10-22 06:52:45][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,563 | MFU: 1.8% [2026-10-22 06:52:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6589 [2026-10-22 06:52:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2283 [2026-10-22 06:52:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8593 [2026-10-22 06:52:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1150 [2026-10-22 06:52:49][INFO][training.pretrain_engine] [Ph2|Step 604] loss=4.9725 | ppl=144.39 | aux=0.02506 | lr=2.97e-04 | gnorm=0.516 | tok/s=29,264 | MFU=7.0% | step=143327ms | tokens=1.436B | mem=4.2/32.7GB | cost=$49.55/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 15:38 (128.8h) | Phase3: SKIPPED | SFT: Oct 27 20:22 (133.5h) | DPO (Final): Oct 27 20:57 (134.1h) [2026-10-22 06:52:49][INFO][training.pretrain_engine] Starting global step 605 (accumulating 32 micro-batches)... [2026-10-22 06:52:49][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 06:53:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0056 [2026-10-22 06:53:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6882 [2026-10-22 06:53:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8349 [2026-10-22 06:53:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0794 [2026-10-22 06:53:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7673 [2026-10-22 06:53:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9392 [2026-10-22 06:53:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8929 [2026-10-22 06:53:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6317 [2026-10-22 06:53:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5837 [2026-10-22 06:53:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3461 [2026-10-22 06:53:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2175 [2026-10-22 06:53:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6492 [2026-10-22 06:54:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9563 [2026-10-22 06:54:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9541 [2026-10-22 06:54:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2321 [2026-10-22 06:54:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9809 [2026-10-22 06:54:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0966 [2026-10-22 06:54:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2237 [2026-10-22 06:54:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1311 [2026-10-22 06:54:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5412 [2026-10-22 06:54:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3300 [2026-10-22 06:54:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6406 [2026-10-22 06:54:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3609 [2026-10-22 06:54:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2893 [2026-10-22 06:54:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0947 [2026-10-22 06:54:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4978 [2026-10-22 06:54:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5405 [2026-10-22 06:54:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6919 [2026-10-22 06:55:08][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,574 | MFU: 1.8% [2026-10-22 06:55:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1188 [2026-10-22 06:55:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7572 [2026-10-22 06:55:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7209 [2026-10-22 06:55:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0841 [2026-10-22 06:55:13][INFO][training.pretrain_engine] [Ph2|Step 605] loss=4.8660 | ppl=129.80 | aux=0.02505 | lr=2.97e-04 | gnorm=0.486 | tok/s=29,261 | MFU=7.0% | step=143340ms | tokens=1.437B | mem=4.2/32.7GB | cost=$49.64/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 15:40 (128.8h) | Phase3: SKIPPED | SFT: Oct 27 20:25 (133.5h) | DPO (Final): Oct 27 20:59 (134.1h) [2026-10-22 06:55:13][INFO][training.pretrain_engine] Starting global step 606 (accumulating 32 micro-batches)... [2026-10-22 06:55:13][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 06:55:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7065 [2026-10-22 06:55:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4706 [2026-10-22 06:55:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9440 [2026-10-22 06:55:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1054 [2026-10-22 06:55:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9582 [2026-10-22 06:55:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5921 [2026-10-22 06:55:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8758 [2026-10-22 06:55:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7993 [2026-10-22 06:56:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=3.9565 [2026-10-22 06:56:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3404 [2026-10-22 06:56:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=3.8478 [2026-10-22 06:56:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=3.3179 [2026-10-22 06:56:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=6.1717 [2026-10-22 06:56:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6054 [2026-10-22 06:56:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8998 [2026-10-22 06:56:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9569 [2026-10-22 06:56:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8411 [2026-10-22 06:56:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7003 [2026-10-22 06:56:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8367 [2026-10-22 06:56:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0190 [2026-10-22 06:57:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7711 [2026-10-22 06:57:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1532 [2026-10-22 06:57:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4662 [2026-10-22 06:57:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8919 [2026-10-22 06:57:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5908 [2026-10-22 06:57:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2446 [2026-10-22 06:57:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1958 [2026-10-22 06:57:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0670 [2026-10-22 06:57:31][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,576 | MFU: 1.8% [2026-10-22 06:57:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0853 [2026-10-22 06:57:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8000 [2026-10-22 06:57:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3206 [2026-10-22 06:57:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0610 [2026-10-22 06:57:37][INFO][training.pretrain_engine] [Ph2|Step 606] loss=4.8253 | ppl=124.62 | aux=0.02506 | lr=2.97e-04 | gnorm=0.543 | tok/s=29,261 | MFU=7.0% | step=143342ms | tokens=1.438B | mem=4.2/32.7GB | cost=$49.72/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 15:42 (128.8h) | Phase3: SKIPPED | SFT: Oct 27 20:27 (133.5h) | DPO (Final): Oct 27 21:01 (134.1h) [2026-10-22 06:57:37][INFO][training.pretrain_engine] Starting global step 607 (accumulating 32 micro-batches)... [2026-10-22 06:57:37][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 06:57:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1567 [2026-10-22 06:57:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8449 [2026-10-22 06:57:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1732 [2026-10-22 06:57:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1277 [2026-10-22 06:58:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9847 [2026-10-22 06:58:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3116 [2026-10-22 06:58:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8260 [2026-10-22 06:58:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9503 [2026-10-22 06:58:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7534 [2026-10-22 06:58:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8140 [2026-10-22 06:58:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4183 [2026-10-22 06:58:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7054 [2026-10-22 06:58:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2694 [2026-10-22 06:58:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7871 [2026-10-22 06:58:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8139 [2026-10-22 06:58:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7904 [2026-10-22 06:59:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4745 [2026-10-22 06:59:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2323 [2026-10-22 06:59:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9849 [2026-10-22 06:59:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2201 [2026-10-22 06:59:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6998 [2026-10-22 06:59:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8818 [2026-10-22 06:59:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5653 [2026-10-22 06:59:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6186 [2026-10-22 06:59:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9752 [2026-10-22 06:59:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6950 [2026-10-22 06:59:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7661 [2026-10-22 06:59:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8114 [2026-10-22 06:59:55][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,580 | MFU: 1.8% [2026-10-22 07:00:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7709 [2026-10-22 07:00:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0496 [2026-10-22 07:00:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3883 [2026-10-22 07:00:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8688 [2026-10-22 07:00:01][INFO][training.pretrain_engine] [Ph2|Step 607] loss=4.8666 | ppl=129.88 | aux=0.02506 | lr=2.97e-04 | gnorm=0.574 | tok/s=29,259 | MFU=7.0% | step=143349ms | tokens=1.439B | mem=4.2/32.7GB | cost=$49.80/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 15:44 (128.7h) | Phase3: SKIPPED | SFT: Oct 27 20:29 (133.5h) | DPO (Final): Oct 27 21:03 (134.1h) [2026-10-22 07:00:01][INFO][training.pretrain_engine] Starting global step 608 (accumulating 32 micro-batches)... [2026-10-22 07:00:01][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 07:00:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2644 [2026-10-22 07:00:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8812 [2026-10-22 07:00:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7889 [2026-10-22 07:00:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0570 [2026-10-22 07:00:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7849 [2026-10-22 07:00:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1354 [2026-10-22 07:00:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8179 [2026-10-22 07:00:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8947 [2026-10-22 07:00:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3679 [2026-10-22 07:00:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8077 [2026-10-22 07:00:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7661 [2026-10-22 07:00:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1929 [2026-10-22 07:01:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0085 [2026-10-22 07:01:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7564 [2026-10-22 07:01:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7803 [2026-10-22 07:01:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8979 [2026-10-22 07:01:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0492 [2026-10-22 07:01:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9641 [2026-10-22 07:01:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5025 [2026-10-22 07:01:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4047 [2026-10-22 07:01:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8716 [2026-10-22 07:01:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8456 [2026-10-22 07:01:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6185 [2026-10-22 07:01:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3215 [2026-10-22 07:02:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1940 [2026-10-22 07:02:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1908 [2026-10-22 07:02:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2837 [2026-10-22 07:02:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7322 [2026-10-22 07:02:19][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,585 | MFU: 1.8% [2026-10-22 07:02:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.0435 [2026-10-22 07:02:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.0002 [2026-10-22 07:02:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3210 [2026-10-22 07:02:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5798 [2026-10-22 07:02:25][INFO][training.pretrain_engine] [Ph2|Step 608] loss=4.7589 | ppl=116.62 | aux=0.02506 | lr=2.97e-04 | gnorm=0.598 | tok/s=29,258 | MFU=7.0% | step=143354ms | tokens=1.440B | mem=4.2/32.7GB | cost=$49.88/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 15:46 (128.7h) | Phase3: SKIPPED | SFT: Oct 27 20:31 (133.5h) | DPO (Final): Oct 27 21:05 (134.1h) [2026-10-22 07:02:25][INFO][training.pretrain_engine] Starting global step 609 (accumulating 32 micro-batches)... [2026-10-22 07:02:25][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 07:02:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4739 [2026-10-22 07:02:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3811 [2026-10-22 07:02:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2932 [2026-10-22 07:02:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6425 [2026-10-22 07:03:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6416 [2026-10-22 07:03:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5724 [2026-10-22 07:03:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6014 [2026-10-22 07:03:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5797 [2026-10-22 07:03:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9787 [2026-10-22 07:03:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6329 [2026-10-22 07:03:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7797 [2026-10-22 07:03:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7179 [2026-10-22 07:03:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0952 [2026-10-22 07:03:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0496 [2026-10-22 07:03:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4342 [2026-10-22 07:03:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3390 [2026-10-22 07:03:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.2995 [2026-10-22 07:03:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0244 [2026-10-22 07:03:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5174 [2026-10-22 07:03:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7140 [2026-10-22 07:04:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9804 [2026-10-22 07:04:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9703 [2026-10-22 07:04:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9528 [2026-10-22 07:04:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9335 [2026-10-22 07:04:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0658 [2026-10-22 07:04:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5518 [2026-10-22 07:04:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8649 [2026-10-22 07:04:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1192 [2026-10-22 07:04:43][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,597 | MFU: 1.8% [2026-10-22 07:04:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1539 [2026-10-22 07:04:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8916 [2026-10-22 07:04:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0617 [2026-10-22 07:04:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9919 [2026-10-22 07:04:49][INFO][training.pretrain_engine] [Ph2|Step 609] loss=4.8982 | ppl=134.05 | aux=0.02505 | lr=2.97e-04 | gnorm=0.703 | tok/s=29,258 | MFU=7.0% | step=143354ms | tokens=1.441B | mem=4.2/32.7GB | cost=$49.97/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 15:48 (128.7h) | Phase3: SKIPPED | SFT: Oct 27 20:33 (133.5h) | DPO (Final): Oct 27 21:07 (134.0h) [2026-10-22 07:04:49][INFO][training.pretrain_engine] Starting global step 610 (accumulating 32 micro-batches)... [2026-10-22 07:04:49][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 07:05:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4256 [2026-10-22 07:05:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0895 [2026-10-22 07:05:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4863 [2026-10-22 07:05:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6075 [2026-10-22 07:05:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0270 [2026-10-22 07:05:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7362 [2026-10-22 07:05:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8930 [2026-10-22 07:05:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5532 [2026-10-22 07:05:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3369 [2026-10-22 07:05:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8164 [2026-10-22 07:05:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8708 [2026-10-22 07:05:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8055 [2026-10-22 07:06:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7854 [2026-10-22 07:06:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5465 [2026-10-22 07:06:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6580 [2026-10-22 07:06:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9619 [2026-10-22 07:06:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2524 [2026-10-22 07:06:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3272 [2026-10-22 07:06:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7104 [2026-10-22 07:06:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2139 [2026-10-22 07:06:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0957 [2026-10-22 07:06:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9903 [2026-10-22 07:06:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0407 [2026-10-22 07:06:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2630 [2026-10-22 07:06:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8851 [2026-10-22 07:06:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8239 [2026-10-22 07:06:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9574 [2026-10-22 07:06:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8252 [2026-10-22 07:07:07][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,589 | MFU: 1.8% [2026-10-22 07:07:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7212 [2026-10-22 07:07:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8951 [2026-10-22 07:07:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4883 [2026-10-22 07:07:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7730 [2026-10-22 07:07:13][INFO][training.pretrain_engine] [Ph2|Step 610] loss=4.8683 | ppl=130.10 | aux=0.02506 | lr=2.97e-04 | gnorm=0.773 | tok/s=29,258 | MFU=7.0% | step=143358ms | tokens=1.442B | mem=4.2/32.7GB | cost=$50.05/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 15:50 (128.7h) | Phase3: SKIPPED | SFT: Oct 27 20:35 (133.5h) | DPO (Final): Oct 27 21:09 (134.0h) [2026-10-22 07:07:13][INFO][training.pretrain_engine] Starting global step 611 (accumulating 32 micro-batches)... [2026-10-22 07:07:13][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 07:07:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0442 [2026-10-22 07:07:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7372 [2026-10-22 07:07:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8402 [2026-10-22 07:07:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6565 [2026-10-22 07:07:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3652 [2026-10-22 07:07:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8136 [2026-10-22 07:07:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6312 [2026-10-22 07:07:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6075 [2026-10-22 07:08:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3814 [2026-10-22 07:08:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=3.9594 [2026-10-22 07:08:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=3.7341 [2026-10-22 07:08:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5128 [2026-10-22 07:08:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1609 [2026-10-22 07:08:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9952 [2026-10-22 07:08:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9046 [2026-10-22 07:08:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2685 [2026-10-22 07:08:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.0286 [2026-10-22 07:08:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4767 [2026-10-22 07:08:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5370 [2026-10-22 07:08:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8876 [2026-10-22 07:08:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3575 [2026-10-22 07:09:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5841 [2026-10-22 07:09:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9791 [2026-10-22 07:09:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3376 [2026-10-22 07:09:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4735 [2026-10-22 07:09:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1647 [2026-10-22 07:09:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5501 [2026-10-22 07:09:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3699 [2026-10-22 07:09:31][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,568 | MFU: 1.8% [2026-10-22 07:09:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5217 [2026-10-22 07:09:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8651 [2026-10-22 07:09:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2346 [2026-10-22 07:09:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1446 [2026-10-22 07:09:36][INFO][training.pretrain_engine] [Ph2|Step 611] loss=4.8230 | ppl=124.33 | aux=0.02506 | lr=2.97e-04 | gnorm=0.691 | tok/s=29,256 | MFU=7.0% | step=143365ms | tokens=1.443B | mem=4.2/32.7GB | cost=$50.13/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 15:52 (128.7h) | Phase3: SKIPPED | SFT: Oct 27 20:37 (133.5h) | DPO (Final): Oct 27 21:11 (134.0h) [2026-10-22 07:09:36][INFO][training.pretrain_engine] Starting global step 612 (accumulating 32 micro-batches)... [2026-10-22 07:09:36][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 07:09:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6307 [2026-10-22 07:09:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1975 [2026-10-22 07:09:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9096 [2026-10-22 07:09:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6732 [2026-10-22 07:10:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9993 [2026-10-22 07:10:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9357 [2026-10-22 07:10:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4164 [2026-10-22 07:10:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0360 [2026-10-22 07:10:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0231 [2026-10-22 07:10:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1135 [2026-10-22 07:10:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1053 [2026-10-22 07:10:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1516 [2026-10-22 07:10:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6999 [2026-10-22 07:10:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7953 [2026-10-22 07:10:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6413 [2026-10-22 07:10:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1863 [2026-10-22 07:11:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.2262 [2026-10-22 07:11:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5417 [2026-10-22 07:11:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5519 [2026-10-22 07:11:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6214 [2026-10-22 07:11:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9922 [2026-10-22 07:11:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8700 [2026-10-22 07:11:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3483 [2026-10-22 07:11:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0576 [2026-10-22 07:11:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4660 [2026-10-22 07:11:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6034 [2026-10-22 07:11:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8075 [2026-10-22 07:11:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=3.7063 [2026-10-22 07:11:55][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,587 | MFU: 1.8% [2026-10-22 07:12:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3896 [2026-10-22 07:12:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2465 [2026-10-22 07:12:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0202 [2026-10-22 07:12:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0710 [2026-10-22 07:12:00][INFO][training.pretrain_engine] [Ph2|Step 612] loss=4.9477 | ppl=140.85 | aux=0.02506 | lr=2.97e-04 | gnorm=0.770 | tok/s=29,255 | MFU=7.0% | step=143372ms | tokens=1.444B | mem=4.2/32.7GB | cost=$50.21/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 15:55 (128.7h) | Phase3: SKIPPED | SFT: Oct 27 20:39 (133.5h) | DPO (Final): Oct 27 21:14 (134.0h) [2026-10-22 07:12:04][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step612 | Tokens: 1.444B | Spent: $50.21 | Remaining: $949.79 [2026-10-22 07:12:04][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step612 [2026-10-22 07:12:04][INFO][training.pretrain_engine] Starting global step 613 (accumulating 32 micro-batches)... [2026-10-22 07:12:04][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 07:12:04][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-22 07:12:04][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-22 07:12:04][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-22 07:12:04][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step612 to Phase-Technologies/forge-500m-pretrain (path: phase2_step612)... [2026-10-22 07:12:04][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-22 07:12:14][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-22 07:12:14][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step612 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-22 07:12:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4547 [2026-10-22 07:12:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9311 [2026-10-22 07:12:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6814 [2026-10-22 07:12:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5757 [2026-10-22 07:12:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7715 [2026-10-22 07:12:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5973 [2026-10-22 07:12:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4347 [2026-10-22 07:12:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9438 [2026-10-22 07:12:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0570 [2026-10-22 07:12:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8398 [2026-10-22 07:12:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2137 [2026-10-22 07:12:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1771 [2026-10-22 07:13:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3334 [2026-10-22 07:13:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6799 [2026-10-22 07:13:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5209 [2026-10-22 07:13:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6217 [2026-10-22 07:13:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7855 [2026-10-22 07:13:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4680 [2026-10-22 07:13:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9678 [2026-10-22 07:13:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6249 [2026-10-22 07:13:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7113 [2026-10-22 07:13:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6349 [2026-10-22 07:13:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0475 [2026-10-22 07:13:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2726 [2026-10-22 07:14:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0562 [2026-10-22 07:14:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1579 [2026-10-22 07:14:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3357 [2026-10-22 07:14:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2461 [2026-10-22 07:14:22][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,578 | MFU: 1.8% [2026-10-22 07:14:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9355 [2026-10-22 07:14:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3519 [2026-10-22 07:14:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9623 [2026-10-22 07:14:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1804 [2026-10-22 07:14:27][INFO][training.pretrain_engine] [Ph2|Step 613] loss=4.9071 | ppl=135.25 | aux=0.02506 | lr=2.97e-04 | gnorm=0.957 | tok/s=29,256 | MFU=7.0% | step=143366ms | tokens=1.445B | mem=4.2/32.7GB | cost=$50.30/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 15:56 (128.7h) | Phase3: SKIPPED | SFT: Oct 27 20:41 (133.4h) | DPO (Final): Oct 27 21:15 (134.0h) [2026-10-22 07:14:27][INFO][training.pretrain_engine] Starting global step 614 (accumulating 32 micro-batches)... [2026-10-22 07:14:27][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 07:14:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0663 [2026-10-22 07:14:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8339 [2026-10-22 07:14:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3569 [2026-10-22 07:14:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3844 [2026-10-22 07:15:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4447 [2026-10-22 07:15:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5984 [2026-10-22 07:15:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0579 [2026-10-22 07:15:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2898 [2026-10-22 07:15:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1968 [2026-10-22 07:15:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3430 [2026-10-22 07:15:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5187 [2026-10-22 07:15:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5099 [2026-10-22 07:15:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9979 [2026-10-22 07:15:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6932 [2026-10-22 07:15:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7174 [2026-10-22 07:15:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8672 [2026-10-22 07:15:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9763 [2026-10-22 07:15:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7880 [2026-10-22 07:15:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9308 [2026-10-22 07:15:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9224 [2026-10-22 07:16:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1089 [2026-10-22 07:16:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8515 [2026-10-22 07:16:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6837 [2026-10-22 07:16:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7924 [2026-10-22 07:16:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9813 [2026-10-22 07:16:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9958 [2026-10-22 07:16:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1079 [2026-10-22 07:16:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0585 [2026-10-22 07:16:46][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,543 | MFU: 1.8% [2026-10-22 07:16:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9883 [2026-10-22 07:16:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2058 [2026-10-22 07:16:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7170 [2026-10-22 07:16:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4036 [2026-10-22 07:16:50][INFO][training.pretrain_engine] [Ph2|Step 614] loss=4.8025 | ppl=121.82 | aux=0.02506 | lr=2.97e-04 | gnorm=1.023 | tok/s=29,253 | MFU=7.0% | step=143379ms | tokens=1.446B | mem=4.2/32.7GB | cost=$50.38/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 15:59 (128.7h) | Phase3: SKIPPED | SFT: Oct 27 20:43 (133.5h) | DPO (Final): Oct 27 21:18 (134.0h) [2026-10-22 07:16:50][INFO][training.pretrain_engine] Starting global step 615 (accumulating 32 micro-batches)... [2026-10-22 07:16:50][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 07:17:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6363 [2026-10-22 07:17:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8531 [2026-10-22 07:17:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6257 [2026-10-22 07:17:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4003 [2026-10-22 07:17:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6898 [2026-10-22 07:17:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.0838 [2026-10-22 07:17:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3445 [2026-10-22 07:17:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4541 [2026-10-22 07:17:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4738 [2026-10-22 07:17:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6403 [2026-10-22 07:17:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8216 [2026-10-22 07:17:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1141 [2026-10-22 07:18:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8272 [2026-10-22 07:18:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4481 [2026-10-22 07:18:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8336 [2026-10-22 07:18:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5441 [2026-10-22 07:18:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3032 [2026-10-22 07:18:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8941 [2026-10-22 07:18:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8191 [2026-10-22 07:18:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8856 [2026-10-22 07:18:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1592 [2026-10-22 07:18:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1205 [2026-10-22 07:18:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6491 [2026-10-22 07:18:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2699 [2026-10-22 07:18:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0175 [2026-10-22 07:18:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2596 [2026-10-22 07:18:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7992 [2026-10-22 07:18:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1193 [2026-10-22 07:19:08][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,602 | MFU: 1.8% [2026-10-22 07:19:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8922 [2026-10-22 07:19:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8639 [2026-10-22 07:19:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9393 [2026-10-22 07:19:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7036 [2026-10-22 07:19:14][INFO][training.pretrain_engine] [Ph2|Step 615] loss=4.8811 | ppl=131.78 | aux=0.02506 | lr=2.97e-04 | gnorm=1.070 | tok/s=29,252 | MFU=7.0% | step=143384ms | tokens=1.447B | mem=4.2/32.7GB | cost=$50.46/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 16:01 (128.7h) | Phase3: SKIPPED | SFT: Oct 27 20:46 (133.4h) | DPO (Final): Oct 27 21:20 (134.0h) [2026-10-22 07:19:14][INFO][training.pretrain_engine] Starting global step 616 (accumulating 32 micro-batches)... [2026-10-22 07:19:14][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 07:19:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0132 [2026-10-22 07:19:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.1653 [2026-10-22 07:19:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2476 [2026-10-22 07:19:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2331 [2026-10-22 07:19:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7751 [2026-10-22 07:19:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7675 [2026-10-22 07:19:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9084 [2026-10-22 07:19:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8664 [2026-10-22 07:20:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5936 [2026-10-22 07:20:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8762 [2026-10-22 07:20:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.0111 [2026-10-22 07:20:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.1284 [2026-10-22 07:20:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0202 [2026-10-22 07:20:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0078 [2026-10-22 07:20:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5491 [2026-10-22 07:20:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1586 [2026-10-22 07:20:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3608 [2026-10-22 07:20:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4545 [2026-10-22 07:20:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5341 [2026-10-22 07:20:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6189 [2026-10-22 07:21:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4957 [2026-10-22 07:21:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0068 [2026-10-22 07:21:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3457 [2026-10-22 07:21:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8506 [2026-10-22 07:21:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8211 [2026-10-22 07:21:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2134 [2026-10-22 07:21:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5024 [2026-10-22 07:21:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7968 [2026-10-22 07:21:32][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,588 | MFU: 1.8% [2026-10-22 07:21:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3425 [2026-10-22 07:21:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7829 [2026-10-22 07:21:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6819 [2026-10-22 07:21:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6980 [2026-10-22 07:21:37][INFO][training.pretrain_engine] [Ph2|Step 616] loss=4.8429 | ppl=126.84 | aux=0.02506 | lr=2.97e-04 | gnorm=0.855 | tok/s=29,252 | MFU=7.0% | step=143387ms | tokens=1.448B | mem=4.2/32.7GB | cost=$50.54/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 16:03 (128.7h) | Phase3: SKIPPED | SFT: Oct 27 20:47 (133.4h) | DPO (Final): Oct 27 21:22 (134.0h) [2026-10-22 07:21:37][INFO][training.pretrain_engine] Starting global step 617 (accumulating 32 micro-batches)... [2026-10-22 07:21:37][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 07:21:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5403 [2026-10-22 07:21:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9782 [2026-10-22 07:21:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0371 [2026-10-22 07:21:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2841 [2026-10-22 07:22:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0938 [2026-10-22 07:22:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4803 [2026-10-22 07:22:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7134 [2026-10-22 07:22:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1074 [2026-10-22 07:22:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9245 [2026-10-22 07:22:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8315 [2026-10-22 07:22:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0171 [2026-10-22 07:22:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7734 [2026-10-22 07:22:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2664 [2026-10-22 07:22:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2512 [2026-10-22 07:22:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7930 [2026-10-22 07:22:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3035 [2026-10-22 07:23:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9135 [2026-10-22 07:23:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9201 [2026-10-22 07:23:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.1630 [2026-10-22 07:23:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9593 [2026-10-22 07:23:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4928 [2026-10-22 07:23:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0695 [2026-10-22 07:23:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4587 [2026-10-22 07:23:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3426 [2026-10-22 07:23:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0705 [2026-10-22 07:23:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1020 [2026-10-22 07:23:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5557 [2026-10-22 07:23:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3819 [2026-10-22 07:23:56][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,581 | MFU: 1.8% [2026-10-22 07:24:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7992 [2026-10-22 07:24:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9668 [2026-10-22 07:24:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9929 [2026-10-22 07:24:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0854 [2026-10-22 07:24:01][INFO][training.pretrain_engine] [Ph2|Step 617] loss=4.9141 | ppl=136.19 | aux=0.02506 | lr=2.97e-04 | gnorm=0.895 | tok/s=29,252 | MFU=7.0% | step=143383ms | tokens=1.449B | mem=4.2/32.7GB | cost=$50.62/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 16:04 (128.7h) | Phase3: SKIPPED | SFT: Oct 27 20:49 (133.4h) | DPO (Final): Oct 27 21:23 (134.0h) [2026-10-22 07:24:01][INFO][training.pretrain_engine] Starting global step 618 (accumulating 32 micro-batches)... [2026-10-22 07:24:01][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 07:24:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0978 [2026-10-22 07:24:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2325 [2026-10-22 07:24:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7355 [2026-10-22 07:24:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0365 [2026-10-22 07:24:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1061 [2026-10-22 07:24:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0430 [2026-10-22 07:24:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1666 [2026-10-22 07:24:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8116 [2026-10-22 07:24:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9257 [2026-10-22 07:24:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1107 [2026-10-22 07:24:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7204 [2026-10-22 07:24:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8544 [2026-10-22 07:25:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2546 [2026-10-22 07:25:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9154 [2026-10-22 07:25:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3640 [2026-10-22 07:25:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0717 [2026-10-22 07:25:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9669 [2026-10-22 07:25:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6927 [2026-10-22 07:25:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9182 [2026-10-22 07:25:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6074 [2026-10-22 07:25:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6319 [2026-10-22 07:25:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1213 [2026-10-22 07:25:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6916 [2026-10-22 07:25:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3835 [2026-10-22 07:26:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9335 [2026-10-22 07:26:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3113 [2026-10-22 07:26:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4516 [2026-10-22 07:26:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6932 [2026-10-22 07:26:19][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,580 | MFU: 1.8% [2026-10-22 07:26:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1429 [2026-10-22 07:26:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0244 [2026-10-22 07:26:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8654 [2026-10-22 07:26:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0156 [2026-10-22 07:26:24][INFO][training.pretrain_engine] [Ph2|Step 618] loss=4.9430 | ppl=140.18 | aux=0.02506 | lr=2.97e-04 | gnorm=0.641 | tok/s=29,253 | MFU=7.0% | step=143381ms | tokens=1.450B | mem=4.2/32.7GB | cost=$50.71/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 16:06 (128.7h) | Phase3: SKIPPED | SFT: Oct 27 20:51 (133.4h) | DPO (Final): Oct 27 21:25 (134.0h) [2026-10-22 07:26:24][INFO][training.pretrain_engine] Starting global step 619 (accumulating 32 micro-batches)... [2026-10-22 07:26:24][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 07:26:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0932 [2026-10-22 07:26:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1905 [2026-10-22 07:26:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1407 [2026-10-22 07:26:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1081 [2026-10-22 07:27:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6964 [2026-10-22 07:27:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8963 [2026-10-22 07:27:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3467 [2026-10-22 07:27:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9108 [2026-10-22 07:27:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2290 [2026-10-22 07:27:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0294 [2026-10-22 07:27:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9399 [2026-10-22 07:27:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1894 [2026-10-22 07:27:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1528 [2026-10-22 07:27:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3899 [2026-10-22 07:27:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1391 [2026-10-22 07:27:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7108 [2026-10-22 07:27:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7364 [2026-10-22 07:27:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8159 [2026-10-22 07:27:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8258 [2026-10-22 07:27:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5109 [2026-10-22 07:28:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7901 [2026-10-22 07:28:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9107 [2026-10-22 07:28:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8933 [2026-10-22 07:28:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7326 [2026-10-22 07:28:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9350 [2026-10-22 07:28:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2638 [2026-10-22 07:28:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4984 [2026-10-22 07:28:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9236 [2026-10-22 07:28:43][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,572 | MFU: 1.8% [2026-10-22 07:28:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9431 [2026-10-22 07:28:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9502 [2026-10-22 07:28:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1850 [2026-10-22 07:28:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7473 [2026-10-22 07:28:47][INFO][training.pretrain_engine] [Ph2|Step 619] loss=4.8589 | ppl=128.88 | aux=0.02507 | lr=2.97e-04 | gnorm=0.688 | tok/s=29,253 | MFU=7.0% | step=143379ms | tokens=1.451B | mem=4.2/32.7GB | cost=$50.79/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 16:08 (128.7h) | Phase3: SKIPPED | SFT: Oct 27 20:52 (133.4h) | DPO (Final): Oct 27 21:27 (134.0h) [2026-10-22 07:28:47][INFO][training.pretrain_engine] Starting global step 620 (accumulating 32 micro-batches)... [2026-10-22 07:28:47][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 07:29:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1498 [2026-10-22 07:29:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0569 [2026-10-22 07:29:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9105 [2026-10-22 07:29:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8298 [2026-10-22 07:29:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7152 [2026-10-22 07:29:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1722 [2026-10-22 07:29:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0434 [2026-10-22 07:29:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1200 [2026-10-22 07:29:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6278 [2026-10-22 07:29:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6773 [2026-10-22 07:29:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7666 [2026-10-22 07:29:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0344 [2026-10-22 07:29:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8800 [2026-10-22 07:29:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1262 [2026-10-22 07:29:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6640 [2026-10-22 07:29:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0676 [2026-10-22 07:30:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.2417 [2026-10-22 07:30:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.1990 [2026-10-22 07:30:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0870 [2026-10-22 07:30:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8591 [2026-10-22 07:30:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0170 [2026-10-22 07:30:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4459 [2026-10-22 07:30:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2437 [2026-10-22 07:30:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4115 [2026-10-22 07:30:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7210 [2026-10-22 07:30:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8171 [2026-10-22 07:30:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0830 [2026-10-22 07:30:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2038 [2026-10-22 07:31:06][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,564 | MFU: 1.8% [2026-10-22 07:31:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3967 [2026-10-22 07:31:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4371 [2026-10-22 07:31:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.1978 [2026-10-22 07:31:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7065 [2026-10-22 07:31:11][INFO][training.pretrain_engine] [Ph2|Step 620] loss=4.8025 | ppl=121.81 | aux=0.02507 | lr=2.97e-04 | gnorm=0.535 | tok/s=29,252 | MFU=7.0% | step=143384ms | tokens=1.452B | mem=4.2/32.7GB | cost=$50.87/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 16:10 (128.6h) | Phase3: SKIPPED | SFT: Oct 27 20:54 (133.4h) | DPO (Final): Oct 27 21:29 (134.0h) [2026-10-22 07:31:11][INFO][training.pretrain_engine] Starting global step 621 (accumulating 32 micro-batches)... [2026-10-22 07:31:11][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 07:31:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4862 [2026-10-22 07:31:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9809 [2026-10-22 07:31:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0317 [2026-10-22 07:31:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8250 [2026-10-22 07:31:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0110 [2026-10-22 07:31:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4009 [2026-10-22 07:31:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5606 [2026-10-22 07:31:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5774 [2026-10-22 07:32:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0424 [2026-10-22 07:32:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6217 [2026-10-22 07:32:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9347 [2026-10-22 07:32:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1515 [2026-10-22 07:32:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3509 [2026-10-22 07:32:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1485 [2026-10-22 07:32:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2591 [2026-10-22 07:32:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5636 [2026-10-22 07:32:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9982 [2026-10-22 07:32:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4061 [2026-10-22 07:32:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8244 [2026-10-22 07:32:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5071 [2026-10-22 07:32:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9078 [2026-10-22 07:32:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6354 [2026-10-22 07:32:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8284 [2026-10-22 07:32:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6761 [2026-10-22 07:33:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1136 [2026-10-22 07:33:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9474 [2026-10-22 07:33:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0807 [2026-10-22 07:33:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5063 [2026-10-22 07:33:30][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,586 | MFU: 1.8% [2026-10-22 07:33:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8812 [2026-10-22 07:33:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6951 [2026-10-22 07:33:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5973 [2026-10-22 07:33:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5775 [2026-10-22 07:33:34][INFO][training.pretrain_engine] [Ph2|Step 621] loss=4.8323 | ppl=125.49 | aux=0.02507 | lr=2.97e-04 | gnorm=0.551 | tok/s=29,253 | MFU=7.0% | step=143381ms | tokens=1.453B | mem=4.2/32.7GB | cost=$50.95/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 16:11 (128.6h) | Phase3: SKIPPED | SFT: Oct 27 20:56 (133.4h) | DPO (Final): Oct 27 21:30 (134.0h) [2026-10-22 07:33:38][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step621 | Tokens: 1.453B | Spent: $50.95 | Remaining: $949.05 [2026-10-22 07:33:38][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step621 [2026-10-22 07:33:38][INFO][training.pretrain_engine] Starting global step 622 (accumulating 32 micro-batches)... [2026-10-22 07:33:38][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 07:33:38][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-22 07:33:38][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-22 07:33:38][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-22 07:33:38][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step621 to Phase-Technologies/forge-500m-pretrain (path: phase2_step621)... [2026-10-22 07:33:38][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-22 07:33:46][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-22 07:33:46][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step621 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-22 07:33:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6105 [2026-10-22 07:33:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9204 [2026-10-22 07:33:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9350 [2026-10-22 07:33:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0169 [2026-10-22 07:34:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6800 [2026-10-22 07:34:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5868 [2026-10-22 07:34:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9589 [2026-10-22 07:34:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5137 [2026-10-22 07:34:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6192 [2026-10-22 07:34:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1019 [2026-10-22 07:34:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0976 [2026-10-22 07:34:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8352 [2026-10-22 07:34:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5348 [2026-10-22 07:34:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2174 [2026-10-22 07:34:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8647 [2026-10-22 07:34:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6501 [2026-10-22 07:35:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8953 [2026-10-22 07:35:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5649 [2026-10-22 07:35:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0840 [2026-10-22 07:35:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0521 [2026-10-22 07:35:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5010 [2026-10-22 07:35:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5578 [2026-10-22 07:35:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6122 [2026-10-22 07:35:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0093 [2026-10-22 07:35:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3287 [2026-10-22 07:35:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5542 [2026-10-22 07:35:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6567 [2026-10-22 07:35:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3145 [2026-10-22 07:35:56][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,594 | MFU: 1.8% [2026-10-22 07:36:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5416 [2026-10-22 07:36:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4503 [2026-10-22 07:36:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3905 [2026-10-22 07:36:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6090 [2026-10-22 07:36:00][INFO][training.pretrain_engine] [Ph2|Step 622] loss=4.9229 | ppl=137.40 | aux=0.02507 | lr=2.97e-04 | gnorm=0.551 | tok/s=29,254 | MFU=7.0% | step=143375ms | tokens=1.454B | mem=4.2/32.7GB | cost=$51.04/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 16:13 (128.6h) | Phase3: SKIPPED | SFT: Oct 27 20:58 (133.4h) | DPO (Final): Oct 27 21:32 (133.9h) [2026-10-22 07:36:00][INFO][training.pretrain_engine] Starting global step 623 (accumulating 32 micro-batches)... [2026-10-22 07:36:00][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 07:36:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1737 [2026-10-22 07:36:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2671 [2026-10-22 07:36:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6299 [2026-10-22 07:36:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4771 [2026-10-22 07:36:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8456 [2026-10-22 07:36:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0580 [2026-10-22 07:36:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9439 [2026-10-22 07:36:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5795 [2026-10-22 07:36:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2217 [2026-10-22 07:36:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0268 [2026-10-22 07:36:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5605 [2026-10-22 07:36:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4798 [2026-10-22 07:37:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2671 [2026-10-22 07:37:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=3.9240 [2026-10-22 07:37:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4365 [2026-10-22 07:37:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9961 [2026-10-22 07:37:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1914 [2026-10-22 07:37:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0217 [2026-10-22 07:37:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2070 [2026-10-22 07:37:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8350 [2026-10-22 07:37:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3805 [2026-10-22 07:37:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9747 [2026-10-22 07:37:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0150 [2026-10-22 07:37:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1294 [2026-10-22 07:38:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3893 [2026-10-22 07:38:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7035 [2026-10-22 07:38:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4224 [2026-10-22 07:38:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.7682 [2026-10-22 07:38:18][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,593 | MFU: 1.8% [2026-10-22 07:38:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4822 [2026-10-22 07:38:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.0658 [2026-10-22 07:38:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9395 [2026-10-22 07:38:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=3.9431 [2026-10-22 07:38:24][INFO][training.pretrain_engine] [Ph2|Step 623] loss=4.9723 | ppl=144.36 | aux=0.02506 | lr=2.97e-04 | gnorm=0.504 | tok/s=29,251 | MFU=7.0% | step=143390ms | tokens=1.455B | mem=4.2/32.7GB | cost=$51.12/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 16:15 (128.6h) | Phase3: SKIPPED | SFT: Oct 27 21:00 (133.4h) | DPO (Final): Oct 27 21:34 (133.9h) [2026-10-22 07:38:24][INFO][training.pretrain_engine] Starting global step 624 (accumulating 32 micro-batches)... [2026-10-22 07:38:24][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 07:38:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7293 [2026-10-22 07:38:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7628 [2026-10-22 07:38:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7723 [2026-10-22 07:38:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5717 [2026-10-22 07:38:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8198 [2026-10-22 07:38:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8277 [2026-10-22 07:39:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6467 [2026-10-22 07:39:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6033 [2026-10-22 07:39:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6398 [2026-10-22 07:39:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3046 [2026-10-22 07:39:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9621 [2026-10-22 07:39:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8903 [2026-10-22 07:39:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7940 [2026-10-22 07:39:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8578 [2026-10-22 07:39:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9308 [2026-10-22 07:39:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2222 [2026-10-22 07:39:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0874 [2026-10-22 07:39:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6048 [2026-10-22 07:39:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4932 [2026-10-22 07:39:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9512 [2026-10-22 07:40:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3112 [2026-10-22 07:40:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7608 [2026-10-22 07:40:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8916 [2026-10-22 07:40:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9010 [2026-10-22 07:40:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1117 [2026-10-22 07:40:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8158 [2026-10-22 07:40:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6177 [2026-10-22 07:40:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9861 [2026-10-22 07:40:42][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,573 | MFU: 1.8% [2026-10-22 07:40:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1247 [2026-10-22 07:40:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5740 [2026-10-22 07:40:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8290 [2026-10-22 07:40:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1170 [2026-10-22 07:40:47][INFO][training.pretrain_engine] [Ph2|Step 624] loss=4.8448 | ppl=127.07 | aux=0.02505 | lr=2.97e-04 | gnorm=0.527 | tok/s=29,250 | MFU=7.0% | step=143397ms | tokens=1.456B | mem=4.2/32.7GB | cost=$51.20/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 16:18 (128.6h) | Phase3: SKIPPED | SFT: Oct 27 21:02 (133.4h) | DPO (Final): Oct 27 21:37 (133.9h) [2026-10-22 07:40:47][INFO][training.pretrain_engine] Starting global step 625 (accumulating 32 micro-batches)... [2026-10-22 07:40:47][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 07:41:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9266 [2026-10-22 07:41:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6278 [2026-10-22 07:41:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4000 [2026-10-22 07:41:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5103 [2026-10-22 07:41:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9127 [2026-10-22 07:41:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0743 [2026-10-22 07:41:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5283 [2026-10-22 07:41:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8247 [2026-10-22 07:41:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2676 [2026-10-22 07:41:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9620 [2026-10-22 07:41:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1946 [2026-10-22 07:41:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9901 [2026-10-22 07:41:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7519 [2026-10-22 07:41:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7091 [2026-10-22 07:41:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7785 [2026-10-22 07:41:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4721 [2026-10-22 07:42:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3923 [2026-10-22 07:42:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7618 [2026-10-22 07:42:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8009 [2026-10-22 07:42:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7043 [2026-10-22 07:42:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2791 [2026-10-22 07:42:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.0955 [2026-10-22 07:42:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2735 [2026-10-22 07:42:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4170 [2026-10-22 07:42:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6709 [2026-10-22 07:42:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4221 [2026-10-22 07:42:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2189 [2026-10-22 07:42:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2749 [2026-10-22 07:43:06][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,582 | MFU: 1.8% [2026-10-22 07:43:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3742 [2026-10-22 07:43:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9937 [2026-10-22 07:43:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9991 [2026-10-22 07:43:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9240 [2026-10-22 07:43:11][INFO][training.pretrain_engine] [Ph2|Step 625] loss=4.7966 | ppl=121.10 | aux=0.02506 | lr=2.97e-04 | gnorm=0.520 | tok/s=29,248 | MFU=7.0% | step=143404ms | tokens=1.458B | mem=4.2/32.7GB | cost=$51.28/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 16:20 (128.6h) | Phase3: SKIPPED | SFT: Oct 27 21:05 (133.4h) | DPO (Final): Oct 27 21:39 (133.9h) [2026-10-22 07:43:11][INFO][training.pretrain_engine] Starting global step 626 (accumulating 32 micro-batches)... [2026-10-22 07:43:11][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 07:43:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2475 [2026-10-22 07:43:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8793 [2026-10-22 07:43:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7597 [2026-10-22 07:43:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3489 [2026-10-22 07:43:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8421 [2026-10-22 07:43:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.0921 [2026-10-22 07:43:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7908 [2026-10-22 07:43:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6732 [2026-10-22 07:44:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6775 [2026-10-22 07:44:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8632 [2026-10-22 07:44:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4151 [2026-10-22 07:44:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2234 [2026-10-22 07:44:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0220 [2026-10-22 07:44:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0846 [2026-10-22 07:44:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0161 [2026-10-22 07:44:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0409 [2026-10-22 07:44:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7946 [2026-10-22 07:44:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6405 [2026-10-22 07:44:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7819 [2026-10-22 07:44:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5086 [2026-10-22 07:44:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9356 [2026-10-22 07:44:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=3.9156 [2026-10-22 07:44:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6920 [2026-10-22 07:44:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4344 [2026-10-22 07:45:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8397 [2026-10-22 07:45:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7082 [2026-10-22 07:45:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1228 [2026-10-22 07:45:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3061 [2026-10-22 07:45:29][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,566 | MFU: 1.8% [2026-10-22 07:45:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3280 [2026-10-22 07:45:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8193 [2026-10-22 07:45:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3713 [2026-10-22 07:45:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4733 [2026-10-22 07:45:34][INFO][training.pretrain_engine] [Ph2|Step 626] loss=4.7205 | ppl=112.23 | aux=0.02507 | lr=2.97e-04 | gnorm=0.527 | tok/s=29,247 | MFU=7.0% | step=143411ms | tokens=1.459B | mem=4.2/32.7GB | cost=$51.36/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 16:22 (128.6h) | Phase3: SKIPPED | SFT: Oct 27 21:07 (133.4h) | DPO (Final): Oct 27 21:41 (133.9h) [2026-10-22 07:45:34][INFO][training.pretrain_engine] Starting global step 627 (accumulating 32 micro-batches)... [2026-10-22 07:45:34][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 07:45:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3708 [2026-10-22 07:45:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4535 [2026-10-22 07:45:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5472 [2026-10-22 07:45:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4090 [2026-10-22 07:46:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5189 [2026-10-22 07:46:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4513 [2026-10-22 07:46:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0475 [2026-10-22 07:46:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2374 [2026-10-22 07:46:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9463 [2026-10-22 07:46:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7259 [2026-10-22 07:46:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8713 [2026-10-22 07:46:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2097 [2026-10-22 07:46:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5366 [2026-10-22 07:46:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9745 [2026-10-22 07:46:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4847 [2026-10-22 07:46:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6425 [2026-10-22 07:47:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9763 [2026-10-22 07:47:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7243 [2026-10-22 07:47:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5278 [2026-10-22 07:47:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6188 [2026-10-22 07:47:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0582 [2026-10-22 07:47:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8259 [2026-10-22 07:47:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1256 [2026-10-22 07:47:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6572 [2026-10-22 07:47:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9505 [2026-10-22 07:47:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9606 [2026-10-22 07:47:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9939 [2026-10-22 07:47:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9595 [2026-10-22 07:47:53][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,584 | MFU: 1.8% [2026-10-22 07:47:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7143 [2026-10-22 07:47:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4465 [2026-10-22 07:47:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7780 [2026-10-22 07:47:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5833 [2026-10-22 07:47:58][INFO][training.pretrain_engine] [Ph2|Step 627] loss=4.8818 | ppl=131.86 | aux=0.02506 | lr=2.97e-04 | gnorm=0.539 | tok/s=29,246 | MFU=7.0% | step=143417ms | tokens=1.460B | mem=4.2/32.7GB | cost=$51.45/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 16:24 (128.6h) | Phase3: SKIPPED | SFT: Oct 27 21:09 (133.4h) | DPO (Final): Oct 27 21:43 (133.9h) [2026-10-22 07:47:58][INFO][training.pretrain_engine] Starting global step 628 (accumulating 32 micro-batches)... [2026-10-22 07:47:58][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 07:48:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9500 [2026-10-22 07:48:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9681 [2026-10-22 07:48:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1241 [2026-10-22 07:48:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9731 [2026-10-22 07:48:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6387 [2026-10-22 07:48:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4534 [2026-10-22 07:48:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5148 [2026-10-22 07:48:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3213 [2026-10-22 07:48:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6170 [2026-10-22 07:48:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4495 [2026-10-22 07:48:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2142 [2026-10-22 07:48:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2391 [2026-10-22 07:49:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4880 [2026-10-22 07:49:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5824 [2026-10-22 07:49:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.0884 [2026-10-22 07:49:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2037 [2026-10-22 07:49:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1008 [2026-10-22 07:49:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7565 [2026-10-22 07:49:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8211 [2026-10-22 07:49:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0576 [2026-10-22 07:49:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4115 [2026-10-22 07:49:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5464 [2026-10-22 07:49:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0386 [2026-10-22 07:49:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.0980 [2026-10-22 07:50:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5345 [2026-10-22 07:50:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9457 [2026-10-22 07:50:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4401 [2026-10-22 07:50:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6902 [2026-10-22 07:50:16][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,610 | MFU: 1.8% [2026-10-22 07:50:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5930 [2026-10-22 07:50:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.1692 [2026-10-22 07:50:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3020 [2026-10-22 07:50:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5948 [2026-10-22 07:50:21][INFO][training.pretrain_engine] [Ph2|Step 628] loss=4.8683 | ppl=130.09 | aux=0.02506 | lr=2.97e-04 | gnorm=0.551 | tok/s=29,244 | MFU=7.0% | step=143423ms | tokens=1.461B | mem=4.2/32.7GB | cost=$51.53/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 16:26 (128.6h) | Phase3: SKIPPED | SFT: Oct 27 21:11 (133.4h) | DPO (Final): Oct 27 21:45 (133.9h) [2026-10-22 07:50:21][INFO][training.pretrain_engine] Starting global step 629 (accumulating 32 micro-batches)... [2026-10-22 07:50:21][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 07:50:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4060 [2026-10-22 07:50:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8453 [2026-10-22 07:50:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5171 [2026-10-22 07:50:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7483 [2026-10-22 07:50:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5377 [2026-10-22 07:50:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8613 [2026-10-22 07:50:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6980 [2026-10-22 07:50:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1343 [2026-10-22 07:51:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6116 [2026-10-22 07:51:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0707 [2026-10-22 07:51:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0720 [2026-10-22 07:51:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5889 [2026-10-22 07:51:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1413 [2026-10-22 07:51:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4681 [2026-10-22 07:51:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7783 [2026-10-22 07:51:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0465 [2026-10-22 07:51:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9851 [2026-10-22 07:51:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8304 [2026-10-22 07:51:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8269 [2026-10-22 07:51:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8154 [2026-10-22 07:52:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3605 [2026-10-22 07:52:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5442 [2026-10-22 07:52:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4264 [2026-10-22 07:52:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4211 [2026-10-22 07:52:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1207 [2026-10-22 07:52:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9334 [2026-10-22 07:52:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4058 [2026-10-22 07:52:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1433 [2026-10-22 07:52:39][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,587 | MFU: 1.8% [2026-10-22 07:52:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5688 [2026-10-22 07:52:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2394 [2026-10-22 07:52:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1089 [2026-10-22 07:52:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2578 [2026-10-22 07:52:44][INFO][training.pretrain_engine] [Ph2|Step 629] loss=4.8253 | ppl=124.62 | aux=0.02505 | lr=2.97e-04 | gnorm=0.516 | tok/s=29,244 | MFU=7.0% | step=143425ms | tokens=1.462B | mem=4.2/32.7GB | cost=$51.61/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 16:28 (128.6h) | Phase3: SKIPPED | SFT: Oct 27 21:13 (133.3h) | DPO (Final): Oct 27 21:47 (133.9h) [2026-10-22 07:52:44][INFO][training.pretrain_engine] Starting global step 630 (accumulating 32 micro-batches)... [2026-10-22 07:52:44][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 07:53:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5504 [2026-10-22 07:53:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8051 [2026-10-22 07:53:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8269 [2026-10-22 07:53:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2969 [2026-10-22 07:53:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1168 [2026-10-22 07:53:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0401 [2026-10-22 07:53:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2487 [2026-10-22 07:53:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4296 [2026-10-22 07:53:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7230 [2026-10-22 07:53:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4958 [2026-10-22 07:53:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7696 [2026-10-22 07:53:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5738 [2026-10-22 07:53:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7264 [2026-10-22 07:53:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5308 [2026-10-22 07:53:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6350 [2026-10-22 07:53:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2424 [2026-10-22 07:54:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1697 [2026-10-22 07:54:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0764 [2026-10-22 07:54:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7402 [2026-10-22 07:54:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2754 [2026-10-22 07:54:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0848 [2026-10-22 07:54:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9267 [2026-10-22 07:54:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6153 [2026-10-22 07:54:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8993 [2026-10-22 07:54:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0550 [2026-10-22 07:54:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7816 [2026-10-22 07:54:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2100 [2026-10-22 07:54:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3489 [2026-10-22 07:55:02][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,613 | MFU: 1.8% [2026-10-22 07:55:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1171 [2026-10-22 07:55:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9048 [2026-10-22 07:55:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0843 [2026-10-22 07:55:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2564 [2026-10-22 07:55:08][INFO][training.pretrain_engine] [Ph2|Step 630] loss=4.8559 | ppl=128.49 | aux=0.02505 | lr=2.97e-04 | gnorm=0.459 | tok/s=29,243 | MFU=7.0% | step=143432ms | tokens=1.463B | mem=4.2/32.7GB | cost=$51.69/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 16:30 (128.6h) | Phase3: SKIPPED | SFT: Oct 27 21:15 (133.3h) | DPO (Final): Oct 27 21:49 (133.9h) [2026-10-22 07:55:11][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step630 | Tokens: 1.463B | Spent: $51.69 | Remaining: $948.31 [2026-10-22 07:55:11][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step630 [2026-10-22 07:55:11][INFO][training.pretrain_engine] Starting global step 631 (accumulating 32 micro-batches)... [2026-10-22 07:55:11][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 07:55:11][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-22 07:55:11][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-22 07:55:12][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-22 07:55:12][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step630 to Phase-Technologies/forge-500m-pretrain (path: phase2_step630)... [2026-10-22 07:55:12][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-22 07:55:20][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-22 07:55:20][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step630 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-22 07:55:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2103 [2026-10-22 07:55:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7354 [2026-10-22 07:55:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0121 [2026-10-22 07:55:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8544 [2026-10-22 07:55:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7982 [2026-10-22 07:55:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7742 [2026-10-22 07:55:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7120 [2026-10-22 07:55:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6929 [2026-10-22 07:56:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0342 [2026-10-22 07:56:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5616 [2026-10-22 07:56:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2831 [2026-10-22 07:56:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7381 [2026-10-22 07:56:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1328 [2026-10-22 07:56:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1612 [2026-10-22 07:56:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8895 [2026-10-22 07:56:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0656 [2026-10-22 07:56:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8896 [2026-10-22 07:56:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6907 [2026-10-22 07:56:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3352 [2026-10-22 07:56:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7434 [2026-10-22 07:56:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8493 [2026-10-22 07:56:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7066 [2026-10-22 07:56:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0053 [2026-10-22 07:56:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6982 [2026-10-22 07:57:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0622 [2026-10-22 07:57:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9816 [2026-10-22 07:57:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7469 [2026-10-22 07:57:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6342 [2026-10-22 07:57:29][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,606 | MFU: 1.8% [2026-10-22 07:57:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9941 [2026-10-22 07:57:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5138 [2026-10-22 07:57:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8373 [2026-10-22 07:57:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5941 [2026-10-22 07:57:34][INFO][training.pretrain_engine] [Ph2|Step 631] loss=4.8568 | ppl=128.62 | aux=0.02506 | lr=2.97e-04 | gnorm=0.562 | tok/s=29,243 | MFU=7.0% | step=143427ms | tokens=1.464B | mem=4.2/32.7GB | cost=$51.78/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 16:32 (128.6h) | Phase3: SKIPPED | SFT: Oct 27 21:17 (133.3h) | DPO (Final): Oct 27 21:51 (133.9h) [2026-10-22 07:57:34][INFO][training.pretrain_engine] Starting global step 632 (accumulating 32 micro-batches)... [2026-10-22 07:57:34][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 07:57:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1139 [2026-10-22 07:57:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8492 [2026-10-22 07:57:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9575 [2026-10-22 07:57:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0418 [2026-10-22 07:58:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0019 [2026-10-22 07:58:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7819 [2026-10-22 07:58:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3400 [2026-10-22 07:58:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6128 [2026-10-22 07:58:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7694 [2026-10-22 07:58:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7357 [2026-10-22 07:58:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1574 [2026-10-22 07:58:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8139 [2026-10-22 07:58:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4212 [2026-10-22 07:58:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3624 [2026-10-22 07:58:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5557 [2026-10-22 07:58:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2651 [2026-10-22 07:59:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1483 [2026-10-22 07:59:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1806 [2026-10-22 07:59:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9857 [2026-10-22 07:59:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9526 [2026-10-22 07:59:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4073 [2026-10-22 07:59:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0914 [2026-10-22 07:59:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1525 [2026-10-22 07:59:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3896 [2026-10-22 07:59:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4353 [2026-10-22 07:59:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4244 [2026-10-22 07:59:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5123 [2026-10-22 07:59:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7576 [2026-10-22 07:59:52][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,590 | MFU: 1.8% [2026-10-22 07:59:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9462 [2026-10-22 07:59:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9544 [2026-10-22 07:59:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1218 [2026-10-22 07:59:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0009 [2026-10-22 07:59:57][INFO][training.pretrain_engine] [Ph2|Step 632] loss=5.0133 | ppl=150.39 | aux=0.02506 | lr=2.97e-04 | gnorm=0.512 | tok/s=29,242 | MFU=7.0% | step=143436ms | tokens=1.465B | mem=4.2/32.7GB | cost=$51.86/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 16:34 (128.6h) | Phase3: SKIPPED | SFT: Oct 27 21:19 (133.3h) | DPO (Final): Oct 27 21:53 (133.9h) [2026-10-22 07:59:57][INFO][training.pretrain_engine] Starting global step 633 (accumulating 32 micro-batches)... [2026-10-22 07:59:57][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 08:00:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8621 [2026-10-22 08:00:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=6.1583 [2026-10-22 08:00:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2716 [2026-10-22 08:00:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4859 [2026-10-22 08:00:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8089 [2026-10-22 08:00:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7694 [2026-10-22 08:00:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0966 [2026-10-22 08:00:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0972 [2026-10-22 08:00:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2788 [2026-10-22 08:00:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1092 [2026-10-22 08:00:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4518 [2026-10-22 08:00:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3491 [2026-10-22 08:01:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5186 [2026-10-22 08:01:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3704 [2026-10-22 08:01:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6176 [2026-10-22 08:01:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3405 [2026-10-22 08:01:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1612 [2026-10-22 08:01:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0450 [2026-10-22 08:01:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1765 [2026-10-22 08:01:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5071 [2026-10-22 08:01:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3136 [2026-10-22 08:01:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1004 [2026-10-22 08:01:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8058 [2026-10-22 08:01:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0760 [2026-10-22 08:02:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5932 [2026-10-22 08:02:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4821 [2026-10-22 08:02:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2448 [2026-10-22 08:02:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5623 [2026-10-22 08:02:15][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,598 | MFU: 1.8% [2026-10-22 08:02:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0207 [2026-10-22 08:02:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0730 [2026-10-22 08:02:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7829 [2026-10-22 08:02:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8178 [2026-10-22 08:02:21][INFO][training.pretrain_engine] [Ph2|Step 633] loss=5.0501 | ppl=156.03 | aux=0.02506 | lr=2.97e-04 | gnorm=0.477 | tok/s=29,241 | MFU=7.0% | step=143438ms | tokens=1.466B | mem=4.2/32.7GB | cost=$51.94/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 16:36 (128.6h) | Phase3: SKIPPED | SFT: Oct 27 21:21 (133.3h) | DPO (Final): Oct 27 21:55 (133.9h) [2026-10-22 08:02:21][INFO][training.pretrain_engine] Starting global step 634 (accumulating 32 micro-batches)... [2026-10-22 08:02:21][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 08:02:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9305 [2026-10-22 08:02:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6891 [2026-10-22 08:02:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1581 [2026-10-22 08:02:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5837 [2026-10-22 08:02:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1373 [2026-10-22 08:02:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9285 [2026-10-22 08:02:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7984 [2026-10-22 08:02:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3126 [2026-10-22 08:03:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5809 [2026-10-22 08:03:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6484 [2026-10-22 08:03:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.1460 [2026-10-22 08:03:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5769 [2026-10-22 08:03:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6271 [2026-10-22 08:03:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3251 [2026-10-22 08:03:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8638 [2026-10-22 08:03:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6044 [2026-10-22 08:03:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4691 [2026-10-22 08:03:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2595 [2026-10-22 08:03:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0425 [2026-10-22 08:03:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3542 [2026-10-22 08:04:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2203 [2026-10-22 08:04:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9166 [2026-10-22 08:04:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8084 [2026-10-22 08:04:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0826 [2026-10-22 08:04:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.0796 [2026-10-22 08:04:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9852 [2026-10-22 08:04:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.0204 [2026-10-22 08:04:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=3.8844 [2026-10-22 08:04:39][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,598 | MFU: 1.8% [2026-10-22 08:04:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5952 [2026-10-22 08:04:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9413 [2026-10-22 08:04:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1961 [2026-10-22 08:04:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0863 [2026-10-22 08:04:45][INFO][training.pretrain_engine] [Ph2|Step 634] loss=4.8814 | ppl=131.82 | aux=0.02506 | lr=2.97e-04 | gnorm=0.543 | tok/s=29,241 | MFU=7.0% | step=143440ms | tokens=1.467B | mem=4.2/32.7GB | cost=$52.02/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 16:38 (128.6h) | Phase3: SKIPPED | SFT: Oct 27 21:23 (133.3h) | DPO (Final): Oct 27 21:57 (133.9h) [2026-10-22 08:04:45][INFO][training.pretrain_engine] Starting global step 635 (accumulating 32 micro-batches)... [2026-10-22 08:04:45][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 08:05:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1286 [2026-10-22 08:05:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6663 [2026-10-22 08:05:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2495 [2026-10-22 08:05:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4569 [2026-10-22 08:05:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2347 [2026-10-22 08:05:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7413 [2026-10-22 08:05:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0507 [2026-10-22 08:05:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1542 [2026-10-22 08:05:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4094 [2026-10-22 08:05:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6782 [2026-10-22 08:05:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7016 [2026-10-22 08:05:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0751 [2026-10-22 08:05:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2563 [2026-10-22 08:05:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2947 [2026-10-22 08:05:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2100 [2026-10-22 08:05:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6608 [2026-10-22 08:06:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6316 [2026-10-22 08:06:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4414 [2026-10-22 08:06:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3118 [2026-10-22 08:06:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9202 [2026-10-22 08:06:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0036 [2026-10-22 08:06:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0355 [2026-10-22 08:06:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8177 [2026-10-22 08:06:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3178 [2026-10-22 08:06:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6861 [2026-10-22 08:06:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5855 [2026-10-22 08:06:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4152 [2026-10-22 08:06:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0738 [2026-10-22 08:07:03][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,589 | MFU: 1.8% [2026-10-22 08:07:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5686 [2026-10-22 08:07:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2984 [2026-10-22 08:07:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5271 [2026-10-22 08:07:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6019 [2026-10-22 08:07:08][INFO][training.pretrain_engine] [Ph2|Step 635] loss=4.8975 | ppl=133.96 | aux=0.02505 | lr=2.97e-04 | gnorm=0.531 | tok/s=29,240 | MFU=7.0% | step=143445ms | tokens=1.468B | mem=4.2/32.7GB | cost=$52.10/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 16:40 (128.6h) | Phase3: SKIPPED | SFT: Oct 27 21:25 (133.3h) | DPO (Final): Oct 27 21:59 (133.9h) [2026-10-22 08:07:08][INFO][training.pretrain_engine] Starting global step 636 (accumulating 32 micro-batches)... [2026-10-22 08:07:08][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 08:07:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9484 [2026-10-22 08:07:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8570 [2026-10-22 08:07:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8609 [2026-10-22 08:07:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0755 [2026-10-22 08:07:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9634 [2026-10-22 08:07:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7030 [2026-10-22 08:07:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8053 [2026-10-22 08:07:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9165 [2026-10-22 08:08:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4376 [2026-10-22 08:08:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5019 [2026-10-22 08:08:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5451 [2026-10-22 08:08:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2581 [2026-10-22 08:08:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3813 [2026-10-22 08:08:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.1151 [2026-10-22 08:08:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2751 [2026-10-22 08:08:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4822 [2026-10-22 08:08:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4136 [2026-10-22 08:08:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.9147 [2026-10-22 08:08:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9029 [2026-10-22 08:08:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9078 [2026-10-22 08:08:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9967 [2026-10-22 08:08:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1032 [2026-10-22 08:08:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0108 [2026-10-22 08:08:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9392 [2026-10-22 08:09:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0116 [2026-10-22 08:09:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9587 [2026-10-22 08:09:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1401 [2026-10-22 08:09:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2520 [2026-10-22 08:09:26][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,584 | MFU: 1.8% [2026-10-22 08:09:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6181 [2026-10-22 08:09:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7440 [2026-10-22 08:09:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4491 [2026-10-22 08:09:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5238 [2026-10-22 08:09:32][INFO][training.pretrain_engine] [Ph2|Step 636] loss=4.9224 | ppl=137.33 | aux=0.02505 | lr=2.97e-04 | gnorm=0.543 | tok/s=29,238 | MFU=7.0% | step=143452ms | tokens=1.469B | mem=4.2/32.7GB | cost=$52.19/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 16:42 (128.6h) | Phase3: SKIPPED | SFT: Oct 27 21:27 (133.3h) | DPO (Final): Oct 27 22:01 (133.9h) [2026-10-22 08:09:32][INFO][training.pretrain_engine] Starting global step 637 (accumulating 32 micro-batches)... [2026-10-22 08:09:32][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 08:09:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6639 [2026-10-22 08:09:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1069 [2026-10-22 08:09:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7898 [2026-10-22 08:09:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8531 [2026-10-22 08:10:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1086 [2026-10-22 08:10:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2175 [2026-10-22 08:10:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3995 [2026-10-22 08:10:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8454 [2026-10-22 08:10:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2881 [2026-10-22 08:10:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9042 [2026-10-22 08:10:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9855 [2026-10-22 08:10:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9185 [2026-10-22 08:10:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2878 [2026-10-22 08:10:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9594 [2026-10-22 08:10:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4946 [2026-10-22 08:10:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0761 [2026-10-22 08:11:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0738 [2026-10-22 08:11:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7375 [2026-10-22 08:11:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1053 [2026-10-22 08:11:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9268 [2026-10-22 08:11:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9404 [2026-10-22 08:11:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6623 [2026-10-22 08:11:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5578 [2026-10-22 08:11:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7593 [2026-10-22 08:11:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0775 [2026-10-22 08:11:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9362 [2026-10-22 08:11:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0329 [2026-10-22 08:11:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5968 [2026-10-22 08:11:50][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,588 | MFU: 1.8% [2026-10-22 08:11:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=3.8664 [2026-10-22 08:11:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4270 [2026-10-22 08:11:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4640 [2026-10-22 08:11:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6078 [2026-10-22 08:11:55][INFO][training.pretrain_engine] [Ph2|Step 637] loss=4.8060 | ppl=122.24 | aux=0.02506 | lr=2.97e-04 | gnorm=0.609 | tok/s=29,237 | MFU=7.0% | step=143459ms | tokens=1.470B | mem=4.2/32.7GB | cost=$52.27/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 16:44 (128.5h) | Phase3: SKIPPED | SFT: Oct 27 21:29 (133.3h) | DPO (Final): Oct 27 22:03 (133.9h) [2026-10-22 08:11:55][INFO][training.pretrain_engine] Starting global step 638 (accumulating 32 micro-batches)... [2026-10-22 08:11:55][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 08:12:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2146 [2026-10-22 08:12:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9821 [2026-10-22 08:12:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9318 [2026-10-22 08:12:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9847 [2026-10-22 08:12:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7271 [2026-10-22 08:12:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2298 [2026-10-22 08:12:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.0092 [2026-10-22 08:12:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8565 [2026-10-22 08:12:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0739 [2026-10-22 08:12:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8579 [2026-10-22 08:12:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2239 [2026-10-22 08:12:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7697 [2026-10-22 08:13:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7318 [2026-10-22 08:13:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9124 [2026-10-22 08:13:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8400 [2026-10-22 08:13:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6821 [2026-10-22 08:13:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6592 [2026-10-22 08:13:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6753 [2026-10-22 08:13:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8198 [2026-10-22 08:13:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1395 [2026-10-22 08:13:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6068 [2026-10-22 08:13:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7250 [2026-10-22 08:13:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.0870 [2026-10-22 08:13:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2554 [2026-10-22 08:14:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2735 [2026-10-22 08:14:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6297 [2026-10-22 08:14:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8247 [2026-10-22 08:14:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2821 [2026-10-22 08:14:13][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,599 | MFU: 1.8% [2026-10-22 08:14:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0102 [2026-10-22 08:14:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7127 [2026-10-22 08:14:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3789 [2026-10-22 08:14:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0534 [2026-10-22 08:14:19][INFO][training.pretrain_engine] [Ph2|Step 638] loss=4.9264 | ppl=137.88 | aux=0.02505 | lr=2.97e-04 | gnorm=0.484 | tok/s=29,236 | MFU=7.0% | step=143463ms | tokens=1.471B | mem=4.2/32.7GB | cost=$52.35/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 16:46 (128.5h) | Phase3: SKIPPED | SFT: Oct 27 21:31 (133.3h) | DPO (Final): Oct 27 22:05 (133.9h) [2026-10-22 08:14:19][INFO][training.pretrain_engine] Starting global step 639 (accumulating 32 micro-batches)... [2026-10-22 08:14:19][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 08:14:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.0163 [2026-10-22 08:14:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6003 [2026-10-22 08:14:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3107 [2026-10-22 08:14:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3440 [2026-10-22 08:14:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0413 [2026-10-22 08:14:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0276 [2026-10-22 08:14:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1108 [2026-10-22 08:14:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9386 [2026-10-22 08:15:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7078 [2026-10-22 08:15:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7336 [2026-10-22 08:15:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0331 [2026-10-22 08:15:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2754 [2026-10-22 08:15:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9798 [2026-10-22 08:15:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9900 [2026-10-22 08:15:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1285 [2026-10-22 08:15:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9110 [2026-10-22 08:15:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4708 [2026-10-22 08:15:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.0581 [2026-10-22 08:15:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3971 [2026-10-22 08:15:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6555 [2026-10-22 08:16:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6639 [2026-10-22 08:16:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7888 [2026-10-22 08:16:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9159 [2026-10-22 08:16:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7206 [2026-10-22 08:16:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3020 [2026-10-22 08:16:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4051 [2026-10-22 08:16:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9558 [2026-10-22 08:16:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5111 [2026-10-22 08:16:38][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,583 | MFU: 1.8% [2026-10-22 08:16:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9194 [2026-10-22 08:16:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7670 [2026-10-22 08:16:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9948 [2026-10-22 08:16:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1348 [2026-10-22 08:16:43][INFO][training.pretrain_engine] [Ph2|Step 639] loss=4.7127 | ppl=111.35 | aux=0.02506 | lr=2.97e-04 | gnorm=0.445 | tok/s=29,235 | MFU=7.0% | step=143469ms | tokens=1.472B | mem=4.2/32.7GB | cost=$52.43/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 16:48 (128.5h) | Phase3: SKIPPED | SFT: Oct 27 21:33 (133.3h) | DPO (Final): Oct 27 22:08 (133.9h) [2026-10-22 08:16:47][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step639 | Tokens: 1.472B | Spent: $52.44 | Remaining: $947.56 [2026-10-22 08:16:47][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step639 [2026-10-22 08:16:47][INFO][training.pretrain_engine] Starting global step 640 (accumulating 32 micro-batches)... [2026-10-22 08:16:47][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 08:16:47][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-22 08:16:47][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-22 08:16:47][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-22 08:16:47][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step639 to Phase-Technologies/forge-500m-pretrain (path: phase2_step639)... [2026-10-22 08:16:47][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-22 08:16:56][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-22 08:16:56][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step639 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-22 08:17:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.0736 [2026-10-22 08:17:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5615 [2026-10-22 08:17:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4317 [2026-10-22 08:17:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3935 [2026-10-22 08:17:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8736 [2026-10-22 08:17:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5186 [2026-10-22 08:17:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6414 [2026-10-22 08:17:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8690 [2026-10-22 08:17:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9719 [2026-10-22 08:17:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5239 [2026-10-22 08:17:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8803 [2026-10-22 08:17:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6445 [2026-10-22 08:17:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5364 [2026-10-22 08:17:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4242 [2026-10-22 08:17:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=3.7943 [2026-10-22 08:17:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4966 [2026-10-22 08:18:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0801 [2026-10-22 08:18:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.2181 [2026-10-22 08:18:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0979 [2026-10-22 08:18:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3807 [2026-10-22 08:18:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5285 [2026-10-22 08:18:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5058 [2026-10-22 08:18:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9212 [2026-10-22 08:18:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3958 [2026-10-22 08:18:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3658 [2026-10-22 08:18:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2515 [2026-10-22 08:18:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0703 [2026-10-22 08:18:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1088 [2026-10-22 08:19:05][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,586 | MFU: 1.8% [2026-10-22 08:19:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8709 [2026-10-22 08:19:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4594 [2026-10-22 08:19:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9606 [2026-10-22 08:19:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6417 [2026-10-22 08:19:09][INFO][training.pretrain_engine] [Ph2|Step 640] loss=4.8176 | ppl=123.66 | aux=0.02505 | lr=2.97e-04 | gnorm=0.512 | tok/s=29,235 | MFU=7.0% | step=143467ms | tokens=1.473B | mem=4.2/32.7GB | cost=$52.52/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 16:50 (128.5h) | Phase3: SKIPPED | SFT: Oct 27 21:35 (133.3h) | DPO (Final): Oct 27 22:09 (133.8h) [2026-10-22 08:19:09][INFO][training.pretrain_engine] Starting global step 641 (accumulating 32 micro-batches)... [2026-10-22 08:19:09][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 08:19:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5873 [2026-10-22 08:19:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9595 [2026-10-22 08:19:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4960 [2026-10-22 08:19:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8218 [2026-10-22 08:19:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2411 [2026-10-22 08:19:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9759 [2026-10-22 08:19:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1468 [2026-10-22 08:19:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5116 [2026-10-22 08:20:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8747 [2026-10-22 08:20:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7696 [2026-10-22 08:20:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6448 [2026-10-22 08:20:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8637 [2026-10-22 08:20:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4589 [2026-10-22 08:20:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4412 [2026-10-22 08:20:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5021 [2026-10-22 08:20:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2828 [2026-10-22 08:20:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8990 [2026-10-22 08:20:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0396 [2026-10-22 08:20:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9136 [2026-10-22 08:20:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3520 [2026-10-22 08:20:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9385 [2026-10-22 08:20:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8261 [2026-10-22 08:20:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0276 [2026-10-22 08:20:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7196 [2026-10-22 08:21:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6954 [2026-10-22 08:21:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7656 [2026-10-22 08:21:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9566 [2026-10-22 08:21:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9713 [2026-10-22 08:21:27][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,602 | MFU: 1.8% [2026-10-22 08:21:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5406 [2026-10-22 08:21:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7897 [2026-10-22 08:21:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7450 [2026-10-22 08:21:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5123 [2026-10-22 08:21:32][INFO][training.pretrain_engine] [Ph2|Step 641] loss=4.8695 | ppl=130.25 | aux=0.02505 | lr=2.97e-04 | gnorm=0.475 | tok/s=29,234 | MFU=7.0% | step=143473ms | tokens=1.474B | mem=4.2/32.7GB | cost=$52.60/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 16:52 (128.5h) | Phase3: SKIPPED | SFT: Oct 27 21:37 (133.3h) | DPO (Final): Oct 27 22:11 (133.8h) [2026-10-22 08:21:32][INFO][training.pretrain_engine] Starting global step 642 (accumulating 32 micro-batches)... [2026-10-22 08:21:32][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 08:21:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6545 [2026-10-22 08:21:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2695 [2026-10-22 08:21:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3590 [2026-10-22 08:21:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6257 [2026-10-22 08:22:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2879 [2026-10-22 08:22:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7772 [2026-10-22 08:22:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9238 [2026-10-22 08:22:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7241 [2026-10-22 08:22:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8325 [2026-10-22 08:22:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8627 [2026-10-22 08:22:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8563 [2026-10-22 08:22:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0684 [2026-10-22 08:22:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8809 [2026-10-22 08:22:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0308 [2026-10-22 08:22:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4555 [2026-10-22 08:22:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9097 [2026-10-22 08:23:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8730 [2026-10-22 08:23:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6611 [2026-10-22 08:23:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9587 [2026-10-22 08:23:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0550 [2026-10-22 08:23:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8819 [2026-10-22 08:23:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7201 [2026-10-22 08:23:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0352 [2026-10-22 08:23:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8064 [2026-10-22 08:23:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8976 [2026-10-22 08:23:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0419 [2026-10-22 08:23:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6887 [2026-10-22 08:23:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7743 [2026-10-22 08:23:51][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,589 | MFU: 1.8% [2026-10-22 08:23:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0995 [2026-10-22 08:23:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9488 [2026-10-22 08:23:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1998 [2026-10-22 08:23:56][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8411 [2026-10-22 08:23:56][INFO][training.pretrain_engine] [Ph2|Step 642] loss=4.8971 | ppl=133.90 | aux=0.02506 | lr=2.97e-04 | gnorm=0.516 | tok/s=29,233 | MFU=7.0% | step=143478ms | tokens=1.475B | mem=4.2/32.7GB | cost=$52.68/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 16:54 (128.5h) | Phase3: SKIPPED | SFT: Oct 27 21:39 (133.3h) | DPO (Final): Oct 27 22:14 (133.8h) [2026-10-22 08:23:56][INFO][training.pretrain_engine] Starting global step 643 (accumulating 32 micro-batches)... [2026-10-22 08:23:56][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 08:24:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1979 [2026-10-22 08:24:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1656 [2026-10-22 08:24:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3135 [2026-10-22 08:24:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0896 [2026-10-22 08:24:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1675 [2026-10-22 08:24:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5265 [2026-10-22 08:24:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0515 [2026-10-22 08:24:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6815 [2026-10-22 08:24:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.0428 [2026-10-22 08:24:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.1348 [2026-10-22 08:24:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2124 [2026-10-22 08:24:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.1182 [2026-10-22 08:25:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8826 [2026-10-22 08:25:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1868 [2026-10-22 08:25:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3523 [2026-10-22 08:25:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1344 [2026-10-22 08:25:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8775 [2026-10-22 08:25:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7639 [2026-10-22 08:25:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7743 [2026-10-22 08:25:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7983 [2026-10-22 08:25:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3686 [2026-10-22 08:25:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6613 [2026-10-22 08:25:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3699 [2026-10-22 08:25:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6224 [2026-10-22 08:26:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0703 [2026-10-22 08:26:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.1755 [2026-10-22 08:26:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6260 [2026-10-22 08:26:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5155 [2026-10-22 08:26:15][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,577 | MFU: 1.8% [2026-10-22 08:26:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6927 [2026-10-22 08:26:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4870 [2026-10-22 08:26:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8340 [2026-10-22 08:26:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8116 [2026-10-22 08:26:19][INFO][training.pretrain_engine] [Ph2|Step 643] loss=4.7453 | ppl=115.04 | aux=0.02506 | lr=2.97e-04 | gnorm=0.621 | tok/s=29,233 | MFU=7.0% | step=143480ms | tokens=1.476B | mem=4.2/32.7GB | cost=$52.76/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 16:56 (128.5h) | Phase3: SKIPPED | SFT: Oct 27 21:41 (133.3h) | DPO (Final): Oct 27 22:15 (133.8h) [2026-10-22 08:26:19][INFO][training.pretrain_engine] Starting global step 644 (accumulating 32 micro-batches)... [2026-10-22 08:26:19][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 08:26:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2556 [2026-10-22 08:26:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1147 [2026-10-22 08:26:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8964 [2026-10-22 08:26:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9963 [2026-10-22 08:26:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0119 [2026-10-22 08:26:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5849 [2026-10-22 08:26:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7122 [2026-10-22 08:26:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9339 [2026-10-22 08:27:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0874 [2026-10-22 08:27:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3092 [2026-10-22 08:27:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9735 [2026-10-22 08:27:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1740 [2026-10-22 08:27:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4938 [2026-10-22 08:27:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0191 [2026-10-22 08:27:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0679 [2026-10-22 08:27:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6710 [2026-10-22 08:27:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3668 [2026-10-22 08:27:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2911 [2026-10-22 08:27:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8668 [2026-10-22 08:27:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2468 [2026-10-22 08:28:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6940 [2026-10-22 08:28:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9409 [2026-10-22 08:28:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0084 [2026-10-22 08:28:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0817 [2026-10-22 08:28:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0139 [2026-10-22 08:28:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7894 [2026-10-22 08:28:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7065 [2026-10-22 08:28:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1752 [2026-10-22 08:28:37][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,601 | MFU: 1.8% [2026-10-22 08:28:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7586 [2026-10-22 08:28:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6446 [2026-10-22 08:28:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3244 [2026-10-22 08:28:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6809 [2026-10-22 08:28:42][INFO][training.pretrain_engine] [Ph2|Step 644] loss=4.8596 | ppl=128.97 | aux=0.02505 | lr=2.97e-04 | gnorm=0.719 | tok/s=29,232 | MFU=7.0% | step=143484ms | tokens=1.477B | mem=4.2/32.7GB | cost=$52.85/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 16:58 (128.5h) | Phase3: SKIPPED | SFT: Oct 27 21:43 (133.3h) | DPO (Final): Oct 27 22:17 (133.8h) [2026-10-22 08:28:42][INFO][training.pretrain_engine] Starting global step 645 (accumulating 32 micro-batches)... [2026-10-22 08:28:42][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 08:29:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8546 [2026-10-22 08:29:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9090 [2026-10-22 08:29:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9810 [2026-10-22 08:29:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1418 [2026-10-22 08:29:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5810 [2026-10-22 08:29:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5780 [2026-10-22 08:29:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3546 [2026-10-22 08:29:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7957 [2026-10-22 08:29:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3251 [2026-10-22 08:29:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3639 [2026-10-22 08:29:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5885 [2026-10-22 08:29:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8463 [2026-10-22 08:29:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9037 [2026-10-22 08:29:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7604 [2026-10-22 08:29:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9838 [2026-10-22 08:29:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9112 [2026-10-22 08:30:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0533 [2026-10-22 08:30:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5414 [2026-10-22 08:30:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7249 [2026-10-22 08:30:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3167 [2026-10-22 08:30:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0924 [2026-10-22 08:30:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0692 [2026-10-22 08:30:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0897 [2026-10-22 08:30:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2200 [2026-10-22 08:30:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7563 [2026-10-22 08:30:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3570 [2026-10-22 08:30:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0474 [2026-10-22 08:30:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6404 [2026-10-22 08:31:01][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,586 | MFU: 1.8% [2026-10-22 08:31:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9412 [2026-10-22 08:31:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8838 [2026-10-22 08:31:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9368 [2026-10-22 08:31:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1870 [2026-10-22 08:31:05][INFO][training.pretrain_engine] [Ph2|Step 645] loss=4.8555 | ppl=128.45 | aux=0.02506 | lr=2.97e-04 | gnorm=0.738 | tok/s=29,233 | MFU=7.0% | step=143481ms | tokens=1.478B | mem=4.2/32.7GB | cost=$52.93/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 17:00 (128.5h) | Phase3: SKIPPED | SFT: Oct 27 21:45 (133.2h) | DPO (Final): Oct 27 22:19 (133.8h) [2026-10-22 08:31:05][INFO][training.pretrain_engine] Starting global step 646 (accumulating 32 micro-batches)... [2026-10-22 08:31:05][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 08:31:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7571 [2026-10-22 08:31:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5350 [2026-10-22 08:31:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9732 [2026-10-22 08:31:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7536 [2026-10-22 08:31:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0686 [2026-10-22 08:31:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0200 [2026-10-22 08:31:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0725 [2026-10-22 08:31:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0855 [2026-10-22 08:31:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9233 [2026-10-22 08:31:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7214 [2026-10-22 08:31:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7508 [2026-10-22 08:31:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0043 [2026-10-22 08:32:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6920 [2026-10-22 08:32:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3355 [2026-10-22 08:32:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5345 [2026-10-22 08:32:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2819 [2026-10-22 08:32:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6251 [2026-10-22 08:32:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7250 [2026-10-22 08:32:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8335 [2026-10-22 08:32:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6989 [2026-10-22 08:32:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4324 [2026-10-22 08:32:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7961 [2026-10-22 08:32:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7572 [2026-10-22 08:32:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2670 [2026-10-22 08:33:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8171 [2026-10-22 08:33:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9136 [2026-10-22 08:33:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5980 [2026-10-22 08:33:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6591 [2026-10-22 08:33:24][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,586 | MFU: 1.8% [2026-10-22 08:33:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1110 [2026-10-22 08:33:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2878 [2026-10-22 08:33:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1281 [2026-10-22 08:33:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0585 [2026-10-22 08:33:28][INFO][training.pretrain_engine] [Ph2|Step 646] loss=4.8342 | ppl=125.74 | aux=0.02505 | lr=2.97e-04 | gnorm=0.848 | tok/s=29,234 | MFU=7.0% | step=143475ms | tokens=1.480B | mem=4.2/32.7GB | cost=$53.01/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 17:01 (128.5h) | Phase3: SKIPPED | SFT: Oct 27 21:46 (133.2h) | DPO (Final): Oct 27 22:21 (133.8h) [2026-10-22 08:33:28][INFO][training.pretrain_engine] Starting global step 647 (accumulating 32 micro-batches)... [2026-10-22 08:33:28][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 08:33:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.0215 [2026-10-22 08:33:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4520 [2026-10-22 08:33:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3507 [2026-10-22 08:33:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7221 [2026-10-22 08:34:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6982 [2026-10-22 08:34:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0305 [2026-10-22 08:34:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4353 [2026-10-22 08:34:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6513 [2026-10-22 08:34:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1516 [2026-10-22 08:34:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9124 [2026-10-22 08:34:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0863 [2026-10-22 08:34:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0053 [2026-10-22 08:34:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0226 [2026-10-22 08:34:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1259 [2026-10-22 08:34:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2570 [2026-10-22 08:34:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0974 [2026-10-22 08:34:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7281 [2026-10-22 08:34:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1387 [2026-10-22 08:34:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7302 [2026-10-22 08:34:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0654 [2026-10-22 08:35:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=3.9755 [2026-10-22 08:35:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.1225 [2026-10-22 08:35:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=3.7711 [2026-10-22 08:35:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2764 [2026-10-22 08:35:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0616 [2026-10-22 08:35:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6486 [2026-10-22 08:35:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2493 [2026-10-22 08:35:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9713 [2026-10-22 08:35:46][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,584 | MFU: 1.8% [2026-10-22 08:35:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0916 [2026-10-22 08:35:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2876 [2026-10-22 08:35:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0341 [2026-10-22 08:35:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3577 [2026-10-22 08:35:51][INFO][training.pretrain_engine] [Ph2|Step 647] loss=4.7593 | ppl=116.66 | aux=0.02506 | lr=2.97e-04 | gnorm=1.102 | tok/s=29,235 | MFU=7.0% | step=143470ms | tokens=1.481B | mem=4.2/32.7GB | cost=$53.09/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 17:03 (128.5h) | Phase3: SKIPPED | SFT: Oct 27 21:48 (133.2h) | DPO (Final): Oct 27 22:22 (133.8h) [2026-10-22 08:35:51][INFO][training.pretrain_engine] Starting global step 648 (accumulating 32 micro-batches)... [2026-10-22 08:35:51][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 08:36:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9489 [2026-10-22 08:36:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4589 [2026-10-22 08:36:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2049 [2026-10-22 08:36:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8657 [2026-10-22 08:36:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1463 [2026-10-22 08:36:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6249 [2026-10-22 08:36:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2428 [2026-10-22 08:36:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2293 [2026-10-22 08:36:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8896 [2026-10-22 08:36:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9096 [2026-10-22 08:36:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1574 [2026-10-22 08:36:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1687 [2026-10-22 08:37:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6939 [2026-10-22 08:37:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8065 [2026-10-22 08:37:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9281 [2026-10-22 08:37:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1419 [2026-10-22 08:37:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9400 [2026-10-22 08:37:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2455 [2026-10-22 08:37:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0049 [2026-10-22 08:37:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6885 [2026-10-22 08:37:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4849 [2026-10-22 08:37:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3342 [2026-10-22 08:37:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7148 [2026-10-22 08:37:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8543 [2026-10-22 08:37:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1039 [2026-10-22 08:37:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8644 [2026-10-22 08:37:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0892 [2026-10-22 08:37:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0531 [2026-10-22 08:38:09][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,585 | MFU: 1.8% [2026-10-22 08:38:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6745 [2026-10-22 08:38:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4097 [2026-10-22 08:38:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5580 [2026-10-22 08:38:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4273 [2026-10-22 08:38:14][INFO][training.pretrain_engine] [Ph2|Step 648] loss=4.9191 | ppl=136.88 | aux=0.02506 | lr=2.97e-04 | gnorm=1.078 | tok/s=29,234 | MFU=7.0% | step=143474ms | tokens=1.482B | mem=4.2/32.7GB | cost=$53.17/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 17:05 (128.5h) | Phase3: SKIPPED | SFT: Oct 27 21:50 (133.2h) | DPO (Final): Oct 27 22:24 (133.8h) [2026-10-22 08:38:17][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step648 | Tokens: 1.482B | Spent: $53.17 | Remaining: $946.83 [2026-10-22 08:38:17][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step648 [2026-10-22 08:38:18][INFO][training.pretrain_engine] Starting global step 649 (accumulating 32 micro-batches)... [2026-10-22 08:38:18][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 08:38:18][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-22 08:38:18][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-22 08:38:18][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-22 08:38:18][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step648 to Phase-Technologies/forge-500m-pretrain (path: phase2_step648)... [2026-10-22 08:38:18][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-22 08:38:27][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-22 08:38:27][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step648 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-22 08:38:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8354 [2026-10-22 08:38:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7914 [2026-10-22 08:38:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0299 [2026-10-22 08:38:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8852 [2026-10-22 08:38:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9255 [2026-10-22 08:38:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9747 [2026-10-22 08:38:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0358 [2026-10-22 08:38:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2580 [2026-10-22 08:39:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1030 [2026-10-22 08:39:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5360 [2026-10-22 08:39:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9029 [2026-10-22 08:39:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7994 [2026-10-22 08:39:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0264 [2026-10-22 08:39:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8301 [2026-10-22 08:39:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9863 [2026-10-22 08:39:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0602 [2026-10-22 08:39:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2512 [2026-10-22 08:39:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3969 [2026-10-22 08:39:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4258 [2026-10-22 08:39:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3956 [2026-10-22 08:40:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8755 [2026-10-22 08:40:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0002 [2026-10-22 08:40:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7536 [2026-10-22 08:40:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1953 [2026-10-22 08:40:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7065 [2026-10-22 08:40:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9186 [2026-10-22 08:40:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7087 [2026-10-22 08:40:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4980 [2026-10-22 08:40:36][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,587 | MFU: 1.8% [2026-10-22 08:40:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7941 [2026-10-22 08:40:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3501 [2026-10-22 08:40:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9727 [2026-10-22 08:40:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4067 [2026-10-22 08:40:40][INFO][training.pretrain_engine] [Ph2|Step 649] loss=5.0289 | ppl=152.77 | aux=0.02507 | lr=2.97e-04 | gnorm=1.117 | tok/s=29,235 | MFU=7.0% | step=143468ms | tokens=1.483B | mem=4.2/32.7GB | cost=$53.26/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 17:06 (128.4h) | Phase3: SKIPPED | SFT: Oct 27 21:51 (133.2h) | DPO (Final): Oct 27 22:26 (133.8h) [2026-10-22 08:40:40][INFO][training.pretrain_engine] Starting global step 650 (accumulating 32 micro-batches)... [2026-10-22 08:40:40][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 08:40:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7374 [2026-10-22 08:40:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7563 [2026-10-22 08:40:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5226 [2026-10-22 08:40:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8773 [2026-10-22 08:41:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1599 [2026-10-22 08:41:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9664 [2026-10-22 08:41:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1328 [2026-10-22 08:41:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5677 [2026-10-22 08:41:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7935 [2026-10-22 08:41:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1337 [2026-10-22 08:41:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8015 [2026-10-22 08:41:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8575 [2026-10-22 08:41:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1809 [2026-10-22 08:41:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5844 [2026-10-22 08:41:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1445 [2026-10-22 08:41:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8620 [2026-10-22 08:42:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0633 [2026-10-22 08:42:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1660 [2026-10-22 08:42:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8494 [2026-10-22 08:42:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2978 [2026-10-22 08:42:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5605 [2026-10-22 08:42:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0170 [2026-10-22 08:42:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6168 [2026-10-22 08:42:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=3.9838 [2026-10-22 08:42:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0756 [2026-10-22 08:42:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3546 [2026-10-22 08:42:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9906 [2026-10-22 08:42:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4112 [2026-10-22 08:42:58][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,588 | MFU: 1.8% [2026-10-22 08:43:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9516 [2026-10-22 08:43:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7384 [2026-10-22 08:43:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7592 [2026-10-22 08:43:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=3.4607 [2026-10-22 08:43:04][INFO][training.pretrain_engine] [Ph2|Step 650] loss=4.8934 | ppl=133.40 | aux=0.02505 | lr=2.97e-04 | gnorm=0.949 | tok/s=29,233 | MFU=7.0% | step=143476ms | tokens=1.484B | mem=4.2/32.7GB | cost=$53.34/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 17:09 (128.4h) | Phase3: SKIPPED | SFT: Oct 27 21:54 (133.2h) | DPO (Final): Oct 27 22:28 (133.8h) [2026-10-22 08:43:04][INFO][training.pretrain_engine] Starting global step 651 (accumulating 32 micro-batches)... [2026-10-22 08:43:04][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 08:43:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3921 [2026-10-22 08:43:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5572 [2026-10-22 08:43:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.4688 [2026-10-22 08:43:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3529 [2026-10-22 08:43:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9775 [2026-10-22 08:43:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6155 [2026-10-22 08:43:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1613 [2026-10-22 08:43:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.7908 [2026-10-22 08:43:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6489 [2026-10-22 08:43:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8159 [2026-10-22 08:43:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2232 [2026-10-22 08:43:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5450 [2026-10-22 08:44:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9945 [2026-10-22 08:44:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3761 [2026-10-22 08:44:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8252 [2026-10-22 08:44:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6063 [2026-10-22 08:44:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8997 [2026-10-22 08:44:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9938 [2026-10-22 08:44:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9739 [2026-10-22 08:44:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0468 [2026-10-22 08:44:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8035 [2026-10-22 08:44:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4853 [2026-10-22 08:44:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8581 [2026-10-22 08:44:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6223 [2026-10-22 08:45:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8847 [2026-10-22 08:45:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8498 [2026-10-22 08:45:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2651 [2026-10-22 08:45:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6297 [2026-10-22 08:45:21][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,619 | MFU: 1.8% [2026-10-22 08:45:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4531 [2026-10-22 08:45:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7286 [2026-10-22 08:45:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4115 [2026-10-22 08:45:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4704 [2026-10-22 08:45:27][INFO][training.pretrain_engine] [Ph2|Step 651] loss=4.8169 | ppl=123.58 | aux=0.02505 | lr=2.97e-04 | gnorm=0.949 | tok/s=29,233 | MFU=7.0% | step=143478ms | tokens=1.485B | mem=4.2/32.7GB | cost=$53.42/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 17:10 (128.4h) | Phase3: SKIPPED | SFT: Oct 27 21:55 (133.2h) | DPO (Final): Oct 27 22:30 (133.7h) [2026-10-22 08:45:27][INFO][training.pretrain_engine] Starting global step 652 (accumulating 32 micro-batches)... [2026-10-22 08:45:27][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 08:45:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6763 [2026-10-22 08:45:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3561 [2026-10-22 08:45:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9054 [2026-10-22 08:45:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6778 [2026-10-22 08:46:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6646 [2026-10-22 08:46:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4767 [2026-10-22 08:46:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0380 [2026-10-22 08:46:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3310 [2026-10-22 08:46:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8479 [2026-10-22 08:46:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7654 [2026-10-22 08:46:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1273 [2026-10-22 08:46:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0575 [2026-10-22 08:46:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6152 [2026-10-22 08:46:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3404 [2026-10-22 08:46:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1147 [2026-10-22 08:46:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1749 [2026-10-22 08:46:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8948 [2026-10-22 08:46:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2387 [2026-10-22 08:46:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9114 [2026-10-22 08:46:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9706 [2026-10-22 08:47:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4203 [2026-10-22 08:47:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6634 [2026-10-22 08:47:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6891 [2026-10-22 08:47:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7576 [2026-10-22 08:47:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8981 [2026-10-22 08:47:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1437 [2026-10-22 08:47:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6748 [2026-10-22 08:47:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8465 [2026-10-22 08:47:45][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,581 | MFU: 1.8% [2026-10-22 08:47:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9953 [2026-10-22 08:47:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0558 [2026-10-22 08:47:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0158 [2026-10-22 08:47:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7462 [2026-10-22 08:47:50][INFO][training.pretrain_engine] [Ph2|Step 652] loss=4.8353 | ppl=125.87 | aux=0.02506 | lr=2.97e-04 | gnorm=0.773 | tok/s=29,233 | MFU=7.0% | step=143477ms | tokens=1.486B | mem=4.2/32.7GB | cost=$53.50/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 17:12 (128.4h) | Phase3: SKIPPED | SFT: Oct 27 21:57 (133.2h) | DPO (Final): Oct 27 22:31 (133.7h) [2026-10-22 08:47:50][INFO][training.pretrain_engine] Starting global step 653 (accumulating 32 micro-batches)... [2026-10-22 08:47:50][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 08:48:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6689 [2026-10-22 08:48:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5351 [2026-10-22 08:48:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3862 [2026-10-22 08:48:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8796 [2026-10-22 08:48:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4436 [2026-10-22 08:48:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0366 [2026-10-22 08:48:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5268 [2026-10-22 08:48:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0023 [2026-10-22 08:48:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4629 [2026-10-22 08:48:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8139 [2026-10-22 08:48:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7051 [2026-10-22 08:48:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5384 [2026-10-22 08:49:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0860 [2026-10-22 08:49:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6742 [2026-10-22 08:49:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9465 [2026-10-22 08:49:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8741 [2026-10-22 08:49:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4662 [2026-10-22 08:49:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3961 [2026-10-22 08:49:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3624 [2026-10-22 08:49:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5500 [2026-10-22 08:49:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9205 [2026-10-22 08:49:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5857 [2026-10-22 08:49:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6602 [2026-10-22 08:49:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0963 [2026-10-22 08:49:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5582 [2026-10-22 08:49:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5033 [2026-10-22 08:49:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1424 [2026-10-22 08:49:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1438 [2026-10-22 08:50:09][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,575 | MFU: 1.8% [2026-10-22 08:50:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8628 [2026-10-22 08:50:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8161 [2026-10-22 08:50:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7180 [2026-10-22 08:50:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0219 [2026-10-22 08:50:14][INFO][training.pretrain_engine] [Ph2|Step 653] loss=4.7828 | ppl=119.44 | aux=0.02505 | lr=2.97e-04 | gnorm=0.750 | tok/s=29,232 | MFU=7.0% | step=143485ms | tokens=1.487B | mem=4.2/32.7GB | cost=$53.58/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 17:14 (128.4h) | Phase3: SKIPPED | SFT: Oct 27 21:59 (133.2h) | DPO (Final): Oct 27 22:34 (133.7h) [2026-10-22 08:50:14][INFO][training.pretrain_engine] Starting global step 654 (accumulating 32 micro-batches)... [2026-10-22 08:50:14][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 08:50:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.0091 [2026-10-22 08:50:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7565 [2026-10-22 08:50:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5135 [2026-10-22 08:50:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.1479 [2026-10-22 08:50:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7176 [2026-10-22 08:50:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5512 [2026-10-22 08:50:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7575 [2026-10-22 08:50:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9085 [2026-10-22 08:51:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6789 [2026-10-22 08:51:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8515 [2026-10-22 08:51:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7181 [2026-10-22 08:51:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4938 [2026-10-22 08:51:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3424 [2026-10-22 08:51:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2663 [2026-10-22 08:51:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9500 [2026-10-22 08:51:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2058 [2026-10-22 08:51:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9253 [2026-10-22 08:51:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5956 [2026-10-22 08:51:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8254 [2026-10-22 08:51:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8857 [2026-10-22 08:52:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0841 [2026-10-22 08:52:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7275 [2026-10-22 08:52:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5928 [2026-10-22 08:52:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7102 [2026-10-22 08:52:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7916 [2026-10-22 08:52:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8674 [2026-10-22 08:52:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0622 [2026-10-22 08:52:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0874 [2026-10-22 08:52:32][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,593 | MFU: 1.8% [2026-10-22 08:52:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8095 [2026-10-22 08:52:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6208 [2026-10-22 08:52:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4070 [2026-10-22 08:52:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9155 [2026-10-22 08:52:37][INFO][training.pretrain_engine] [Ph2|Step 654] loss=4.7288 | ppl=113.16 | aux=0.02506 | lr=2.97e-04 | gnorm=0.711 | tok/s=29,231 | MFU=7.0% | step=143490ms | tokens=1.488B | mem=4.2/32.7GB | cost=$53.67/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 17:16 (128.4h) | Phase3: SKIPPED | SFT: Oct 27 22:01 (133.2h) | DPO (Final): Oct 27 22:36 (133.7h) [2026-10-22 08:52:37][INFO][training.pretrain_engine] Starting global step 655 (accumulating 32 micro-batches)... [2026-10-22 08:52:37][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 08:52:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0103 [2026-10-22 08:52:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.8672 [2026-10-22 08:52:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6300 [2026-10-22 08:52:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.9087 [2026-10-22 08:53:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9778 [2026-10-22 08:53:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1149 [2026-10-22 08:53:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9879 [2026-10-22 08:53:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2529 [2026-10-22 08:53:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3918 [2026-10-22 08:53:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8912 [2026-10-22 08:53:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4025 [2026-10-22 08:53:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1764 [2026-10-22 08:53:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=3.6190 [2026-10-22 08:53:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.1212 [2026-10-22 08:53:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3219 [2026-10-22 08:53:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.0464 [2026-10-22 08:54:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8137 [2026-10-22 08:54:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1537 [2026-10-22 08:54:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4638 [2026-10-22 08:54:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0450 [2026-10-22 08:54:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1558 [2026-10-22 08:54:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5103 [2026-10-22 08:54:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4458 [2026-10-22 08:54:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8225 [2026-10-22 08:54:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6092 [2026-10-22 08:54:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7963 [2026-10-22 08:54:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9959 [2026-10-22 08:54:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9887 [2026-10-22 08:54:56][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,571 | MFU: 1.8% [2026-10-22 08:55:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8805 [2026-10-22 08:55:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1393 [2026-10-22 08:55:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1337 [2026-10-22 08:55:01][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6350 [2026-10-22 08:55:01][INFO][training.pretrain_engine] [Ph2|Step 655] loss=4.8002 | ppl=121.53 | aux=0.02506 | lr=2.97e-04 | gnorm=0.605 | tok/s=29,231 | MFU=7.0% | step=143490ms | tokens=1.489B | mem=4.2/32.7GB | cost=$53.75/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 17:18 (128.4h) | Phase3: SKIPPED | SFT: Oct 27 22:03 (133.1h) | DPO (Final): Oct 27 22:38 (133.7h) [2026-10-22 08:55:01][INFO][training.pretrain_engine] Starting global step 656 (accumulating 32 micro-batches)... [2026-10-22 08:55:01][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 08:55:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2554 [2026-10-22 08:55:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0050 [2026-10-22 08:55:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9483 [2026-10-22 08:55:19][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2301 [2026-10-22 08:55:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7859 [2026-10-22 08:55:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9215 [2026-10-22 08:55:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1029 [2026-10-22 08:55:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2592 [2026-10-22 08:55:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7975 [2026-10-22 08:55:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2812 [2026-10-22 08:55:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4752 [2026-10-22 08:55:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4959 [2026-10-22 08:56:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6231 [2026-10-22 08:56:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9045 [2026-10-22 08:56:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7231 [2026-10-22 08:56:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6056 [2026-10-22 08:56:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9512 [2026-10-22 08:56:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6759 [2026-10-22 08:56:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9676 [2026-10-22 08:56:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9603 [2026-10-22 08:56:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6630 [2026-10-22 08:56:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1919 [2026-10-22 08:56:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3758 [2026-10-22 08:56:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1937 [2026-10-22 08:57:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5856 [2026-10-22 08:57:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0814 [2026-10-22 08:57:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6222 [2026-10-22 08:57:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1467 [2026-10-22 08:57:19][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,584 | MFU: 1.8% [2026-10-22 08:57:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7734 [2026-10-22 08:57:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7895 [2026-10-22 08:57:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9869 [2026-10-22 08:57:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0194 [2026-10-22 08:57:24][INFO][training.pretrain_engine] [Ph2|Step 656] loss=4.8500 | ppl=127.75 | aux=0.02505 | lr=2.97e-04 | gnorm=0.648 | tok/s=29,232 | MFU=7.0% | step=143481ms | tokens=1.490B | mem=4.2/32.7GB | cost=$53.83/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 17:20 (128.4h) | Phase3: SKIPPED | SFT: Oct 27 22:05 (133.1h) | DPO (Final): Oct 27 22:39 (133.7h) [2026-10-22 08:57:24][INFO][training.pretrain_engine] Starting global step 657 (accumulating 32 micro-batches)... [2026-10-22 08:57:24][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 08:57:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9478 [2026-10-22 08:57:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4609 [2026-10-22 08:57:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1754 [2026-10-22 08:57:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7821 [2026-10-22 08:57:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4950 [2026-10-22 08:57:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6959 [2026-10-22 08:57:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6435 [2026-10-22 08:57:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0352 [2026-10-22 08:58:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6122 [2026-10-22 08:58:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5882 [2026-10-22 08:58:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8079 [2026-10-22 08:58:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5724 [2026-10-22 08:58:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7732 [2026-10-22 08:58:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6456 [2026-10-22 08:58:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6085 [2026-10-22 08:58:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5691 [2026-10-22 08:58:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6845 [2026-10-22 08:58:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9003 [2026-10-22 08:58:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4180 [2026-10-22 08:58:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1556 [2026-10-22 08:59:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1386 [2026-10-22 08:59:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1214 [2026-10-22 08:59:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0616 [2026-10-22 08:59:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7931 [2026-10-22 08:59:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0548 [2026-10-22 08:59:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9820 [2026-10-22 08:59:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5693 [2026-10-22 08:59:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7185 [2026-10-22 08:59:41][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,609 | MFU: 1.8% [2026-10-22 08:59:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1370 [2026-10-22 08:59:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5634 [2026-10-22 08:59:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6608 [2026-10-22 08:59:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8189 [2026-10-22 08:59:46][INFO][training.pretrain_engine] [Ph2|Step 657] loss=4.7799 | ppl=119.10 | aux=0.02506 | lr=2.97e-04 | gnorm=0.590 | tok/s=29,236 | MFU=7.0% | step=143466ms | tokens=1.491B | mem=4.2/32.7GB | cost=$53.91/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 17:20 (128.4h) | Phase3: SKIPPED | SFT: Oct 27 22:05 (133.1h) | DPO (Final): Oct 27 22:40 (133.7h) [2026-10-22 08:59:50][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step657 | Tokens: 1.491B | Spent: $53.91 | Remaining: $946.09 [2026-10-22 08:59:50][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step657 [2026-10-22 08:59:50][INFO][training.pretrain_engine] Starting global step 658 (accumulating 32 micro-batches)... [2026-10-22 08:59:50][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 08:59:50][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-22 08:59:50][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-22 08:59:50][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-22 08:59:50][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step657 to Phase-Technologies/forge-500m-pretrain (path: phase2_step657)... [2026-10-22 08:59:50][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-22 09:00:00][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-22 09:00:00][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step657 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-22 09:00:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2340 [2026-10-22 09:00:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8419 [2026-10-22 09:00:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7549 [2026-10-22 09:00:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8600 [2026-10-22 09:00:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3890 [2026-10-22 09:00:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3561 [2026-10-22 09:00:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.7149 [2026-10-22 09:00:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2477 [2026-10-22 09:00:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9942 [2026-10-22 09:00:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4092 [2026-10-22 09:00:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3899 [2026-10-22 09:00:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=3.8298 [2026-10-22 09:00:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6748 [2026-10-22 09:00:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1139 [2026-10-22 09:00:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2120 [2026-10-22 09:01:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6702 [2026-10-22 09:01:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8831 [2026-10-22 09:01:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7428 [2026-10-22 09:01:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8892 [2026-10-22 09:01:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9562 [2026-10-22 09:01:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0309 [2026-10-22 09:01:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2630 [2026-10-22 09:01:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0776 [2026-10-22 09:01:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9206 [2026-10-22 09:01:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1131 [2026-10-22 09:01:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7344 [2026-10-22 09:01:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0257 [2026-10-22 09:01:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7750 [2026-10-22 09:02:07][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,612 | MFU: 1.8% [2026-10-22 09:02:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0236 [2026-10-22 09:02:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8194 [2026-10-22 09:02:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9032 [2026-10-22 09:02:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2363 [2026-10-22 09:02:12][INFO][training.pretrain_engine] [Ph2|Step 658] loss=4.8430 | ppl=126.84 | aux=0.02506 | lr=2.97e-04 | gnorm=0.559 | tok/s=29,239 | MFU=7.0% | step=143451ms | tokens=1.492B | mem=4.2/32.7GB | cost=$54.00/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 17:22 (128.3h) | Phase3: SKIPPED | SFT: Oct 27 22:07 (133.1h) | DPO (Final): Oct 27 22:41 (133.7h) [2026-10-22 09:02:12][INFO][training.pretrain_engine] Starting global step 659 (accumulating 32 micro-batches)... [2026-10-22 09:02:12][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 09:02:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7853 [2026-10-22 09:02:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1482 [2026-10-22 09:02:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8016 [2026-10-22 09:02:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5763 [2026-10-22 09:02:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.0316 [2026-10-22 09:02:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1698 [2026-10-22 09:02:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5735 [2026-10-22 09:02:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.8113 [2026-10-22 09:03:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9706 [2026-10-22 09:03:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1093 [2026-10-22 09:03:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0038 [2026-10-22 09:03:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3902 [2026-10-22 09:03:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.1232 [2026-10-22 09:03:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5520 [2026-10-22 09:03:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7808 [2026-10-22 09:03:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3060 [2026-10-22 09:03:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2600 [2026-10-22 09:03:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1563 [2026-10-22 09:03:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3192 [2026-10-22 09:03:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9244 [2026-10-22 09:03:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7573 [2026-10-22 09:03:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9054 [2026-10-22 09:03:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8196 [2026-10-22 09:03:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8344 [2026-10-22 09:04:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8285 [2026-10-22 09:04:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7086 [2026-10-22 09:04:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9909 [2026-10-22 09:04:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1647 [2026-10-22 09:04:30][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,599 | MFU: 1.8% [2026-10-22 09:04:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7655 [2026-10-22 09:04:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=3.8352 [2026-10-22 09:04:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.0203 [2026-10-22 09:04:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7688 [2026-10-22 09:04:35][INFO][training.pretrain_engine] [Ph2|Step 659] loss=4.6996 | ppl=109.91 | aux=0.02506 | lr=2.97e-04 | gnorm=0.531 | tok/s=29,237 | MFU=7.0% | step=143460ms | tokens=1.493B | mem=4.2/32.7GB | cost=$54.08/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 17:24 (128.3h) | Phase3: SKIPPED | SFT: Oct 27 22:09 (133.1h) | DPO (Final): Oct 27 22:43 (133.6h) [2026-10-22 09:04:35][INFO][training.pretrain_engine] Starting global step 660 (accumulating 32 micro-batches)... [2026-10-22 09:04:35][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 09:04:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9391 [2026-10-22 09:04:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0776 [2026-10-22 09:04:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9017 [2026-10-22 09:04:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3212 [2026-10-22 09:05:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0356 [2026-10-22 09:05:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5246 [2026-10-22 09:05:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4475 [2026-10-22 09:05:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0062 [2026-10-22 09:05:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1478 [2026-10-22 09:05:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0035 [2026-10-22 09:05:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2317 [2026-10-22 09:05:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8145 [2026-10-22 09:05:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.1425 [2026-10-22 09:05:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6090 [2026-10-22 09:05:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2512 [2026-10-22 09:05:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4565 [2026-10-22 09:06:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=3.9386 [2026-10-22 09:06:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4796 [2026-10-22 09:06:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3523 [2026-10-22 09:06:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7097 [2026-10-22 09:06:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0353 [2026-10-22 09:06:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9965 [2026-10-22 09:06:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5958 [2026-10-22 09:06:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2354 [2026-10-22 09:06:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6224 [2026-10-22 09:06:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=3.8257 [2026-10-22 09:06:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3398 [2026-10-22 09:06:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8620 [2026-10-22 09:06:53][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,594 | MFU: 1.8% [2026-10-22 09:06:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0307 [2026-10-22 09:06:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9009 [2026-10-22 09:06:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9979 [2026-10-22 09:06:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1722 [2026-10-22 09:06:58][INFO][training.pretrain_engine] [Ph2|Step 660] loss=4.7481 | ppl=115.36 | aux=0.02505 | lr=2.97e-04 | gnorm=0.486 | tok/s=29,238 | MFU=7.0% | step=143455ms | tokens=1.494B | mem=4.2/32.7GB | cost=$54.16/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 17:25 (128.3h) | Phase3: SKIPPED | SFT: Oct 27 22:10 (133.1h) | DPO (Final): Oct 27 22:45 (133.6h) [2026-10-22 09:06:58][INFO][training.pretrain_engine] Starting global step 661 (accumulating 32 micro-batches)... [2026-10-22 09:06:58][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 09:07:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.8006 [2026-10-22 09:07:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7028 [2026-10-22 09:07:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9781 [2026-10-22 09:07:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1006 [2026-10-22 09:07:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3466 [2026-10-22 09:07:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4958 [2026-10-22 09:07:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1470 [2026-10-22 09:07:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2239 [2026-10-22 09:07:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5095 [2026-10-22 09:07:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5667 [2026-10-22 09:07:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3993 [2026-10-22 09:07:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7135 [2026-10-22 09:08:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7395 [2026-10-22 09:08:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8688 [2026-10-22 09:08:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3329 [2026-10-22 09:08:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3826 [2026-10-22 09:08:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4355 [2026-10-22 09:08:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4436 [2026-10-22 09:08:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3676 [2026-10-22 09:08:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9648 [2026-10-22 09:08:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9654 [2026-10-22 09:08:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9456 [2026-10-22 09:08:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9068 [2026-10-22 09:08:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8310 [2026-10-22 09:09:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9400 [2026-10-22 09:09:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3676 [2026-10-22 09:09:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7764 [2026-10-22 09:09:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5248 [2026-10-22 09:09:16][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,624 | MFU: 1.8% [2026-10-22 09:09:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8990 [2026-10-22 09:09:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5793 [2026-10-22 09:09:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0639 [2026-10-22 09:09:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6006 [2026-10-22 09:09:22][INFO][training.pretrain_engine] [Ph2|Step 661] loss=4.7909 | ppl=120.41 | aux=0.02506 | lr=2.97e-04 | gnorm=0.496 | tok/s=29,239 | MFU=7.0% | step=143450ms | tokens=1.495B | mem=4.2/32.7GB | cost=$54.24/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 17:27 (128.3h) | Phase3: SKIPPED | SFT: Oct 27 22:12 (133.0h) | DPO (Final): Oct 27 22:46 (133.6h) [2026-10-22 09:09:22][INFO][training.pretrain_engine] Starting global step 662 (accumulating 32 micro-batches)... [2026-10-22 09:09:22][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 09:09:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9168 [2026-10-22 09:09:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7689 [2026-10-22 09:09:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8277 [2026-10-22 09:09:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7192 [2026-10-22 09:09:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3165 [2026-10-22 09:09:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.4414 [2026-10-22 09:09:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8397 [2026-10-22 09:09:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.0009 [2026-10-22 09:10:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8472 [2026-10-22 09:10:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1464 [2026-10-22 09:10:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2958 [2026-10-22 09:10:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4772 [2026-10-22 09:10:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.1615 [2026-10-22 09:10:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0155 [2026-10-22 09:10:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5592 [2026-10-22 09:10:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5925 [2026-10-22 09:10:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9292 [2026-10-22 09:10:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0802 [2026-10-22 09:10:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3223 [2026-10-22 09:10:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1692 [2026-10-22 09:11:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7101 [2026-10-22 09:11:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8785 [2026-10-22 09:11:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8691 [2026-10-22 09:11:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6551 [2026-10-22 09:11:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2066 [2026-10-22 09:11:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7305 [2026-10-22 09:11:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1195 [2026-10-22 09:11:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9733 [2026-10-22 09:11:40][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,600 | MFU: 1.8% [2026-10-22 09:11:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.0383 [2026-10-22 09:11:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9523 [2026-10-22 09:11:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7446 [2026-10-22 09:11:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7741 [2026-10-22 09:11:45][INFO][training.pretrain_engine] [Ph2|Step 662] loss=4.8496 | ppl=127.69 | aux=0.02505 | lr=2.97e-04 | gnorm=0.500 | tok/s=29,239 | MFU=7.0% | step=143447ms | tokens=1.496B | mem=4.2/32.7GB | cost=$54.32/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 17:28 (128.3h) | Phase3: SKIPPED | SFT: Oct 27 22:13 (133.0h) | DPO (Final): Oct 27 22:48 (133.6h) [2026-10-22 09:11:45][INFO][training.pretrain_engine] Starting global step 663 (accumulating 32 micro-batches)... [2026-10-22 09:11:45][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 09:12:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0388 [2026-10-22 09:12:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4997 [2026-10-22 09:12:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0195 [2026-10-22 09:12:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3137 [2026-10-22 09:12:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9284 [2026-10-22 09:12:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8300 [2026-10-22 09:12:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1001 [2026-10-22 09:12:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7076 [2026-10-22 09:12:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1905 [2026-10-22 09:12:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2356 [2026-10-22 09:12:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0517 [2026-10-22 09:12:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4180 [2026-10-22 09:12:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5434 [2026-10-22 09:12:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8970 [2026-10-22 09:12:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5768 [2026-10-22 09:12:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5592 [2026-10-22 09:13:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1878 [2026-10-22 09:13:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0817 [2026-10-22 09:13:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3405 [2026-10-22 09:13:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8131 [2026-10-22 09:13:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5272 [2026-10-22 09:13:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1096 [2026-10-22 09:13:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9573 [2026-10-22 09:13:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5867 [2026-10-22 09:13:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4698 [2026-10-22 09:13:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4163 [2026-10-22 09:13:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5419 [2026-10-22 09:13:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.1771 [2026-10-22 09:14:03][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,596 | MFU: 1.8% [2026-10-22 09:14:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9316 [2026-10-22 09:14:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1305 [2026-10-22 09:14:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8844 [2026-10-22 09:14:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2034 [2026-10-22 09:14:09][INFO][training.pretrain_engine] [Ph2|Step 663] loss=4.7501 | ppl=115.60 | aux=0.02506 | lr=2.97e-04 | gnorm=0.490 | tok/s=29,240 | MFU=7.0% | step=143445ms | tokens=1.497B | mem=4.2/32.7GB | cost=$54.41/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 17:30 (128.3h) | Phase3: SKIPPED | SFT: Oct 27 22:15 (133.0h) | DPO (Final): Oct 27 22:49 (133.6h) [2026-10-22 09:14:09][INFO][training.pretrain_engine] Starting global step 664 (accumulating 32 micro-batches)... [2026-10-22 09:14:09][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 09:14:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4894 [2026-10-22 09:14:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2042 [2026-10-22 09:14:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8987 [2026-10-22 09:14:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0223 [2026-10-22 09:14:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9120 [2026-10-22 09:14:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8045 [2026-10-22 09:14:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6592 [2026-10-22 09:14:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7449 [2026-10-22 09:15:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8627 [2026-10-22 09:15:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3074 [2026-10-22 09:15:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1066 [2026-10-22 09:15:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9601 [2026-10-22 09:15:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7464 [2026-10-22 09:15:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6149 [2026-10-22 09:15:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8477 [2026-10-22 09:15:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5689 [2026-10-22 09:15:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8471 [2026-10-22 09:15:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7368 [2026-10-22 09:15:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.2390 [2026-10-22 09:15:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8705 [2026-10-22 09:15:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0816 [2026-10-22 09:15:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4242 [2026-10-22 09:15:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2773 [2026-10-22 09:15:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9402 [2026-10-22 09:16:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9594 [2026-10-22 09:16:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9481 [2026-10-22 09:16:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8716 [2026-10-22 09:16:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8951 [2026-10-22 09:16:27][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,609 | MFU: 1.8% [2026-10-22 09:16:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3107 [2026-10-22 09:16:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1472 [2026-10-22 09:16:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=3.9368 [2026-10-22 09:16:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6508 [2026-10-22 09:16:32][INFO][training.pretrain_engine] [Ph2|Step 664] loss=4.8836 | ppl=132.11 | aux=0.02506 | lr=2.97e-04 | gnorm=0.479 | tok/s=29,241 | MFU=7.0% | step=143442ms | tokens=1.498B | mem=4.2/32.7GB | cost=$54.49/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 17:32 (128.3h) | Phase3: SKIPPED | SFT: Oct 27 22:17 (133.0h) | DPO (Final): Oct 27 22:51 (133.6h) [2026-10-22 09:16:32][INFO][training.pretrain_engine] Starting global step 665 (accumulating 32 micro-batches)... [2026-10-22 09:16:32][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 09:16:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.1665 [2026-10-22 09:16:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.0637 [2026-10-22 09:16:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2924 [2026-10-22 09:16:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4802 [2026-10-22 09:17:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2958 [2026-10-22 09:17:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6061 [2026-10-22 09:17:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0133 [2026-10-22 09:17:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8640 [2026-10-22 09:17:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1384 [2026-10-22 09:17:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8866 [2026-10-22 09:17:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2639 [2026-10-22 09:17:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0567 [2026-10-22 09:17:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8927 [2026-10-22 09:17:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2387 [2026-10-22 09:17:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7381 [2026-10-22 09:17:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3309 [2026-10-22 09:18:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4117 [2026-10-22 09:18:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9091 [2026-10-22 09:18:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7628 [2026-10-22 09:18:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5948 [2026-10-22 09:18:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7404 [2026-10-22 09:18:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0944 [2026-10-22 09:18:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8491 [2026-10-22 09:18:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5633 [2026-10-22 09:18:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2669 [2026-10-22 09:18:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5661 [2026-10-22 09:18:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=3.9641 [2026-10-22 09:18:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9007 [2026-10-22 09:18:50][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,587 | MFU: 1.8% [2026-10-22 09:18:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2308 [2026-10-22 09:18:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6915 [2026-10-22 09:18:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9495 [2026-10-22 09:18:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9525 [2026-10-22 09:18:55][INFO][training.pretrain_engine] [Ph2|Step 665] loss=4.7939 | ppl=120.77 | aux=0.02506 | lr=2.97e-04 | gnorm=0.508 | tok/s=29,242 | MFU=7.0% | step=143434ms | tokens=1.499B | mem=4.2/32.7GB | cost=$54.57/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 17:33 (128.2h) | Phase3: SKIPPED | SFT: Oct 27 22:18 (133.0h) | DPO (Final): Oct 27 22:52 (133.6h) [2026-10-22 09:18:55][INFO][training.pretrain_engine] Starting global step 666 (accumulating 32 micro-batches)... [2026-10-22 09:18:55][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 09:19:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7128 [2026-10-22 09:19:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1535 [2026-10-22 09:19:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7119 [2026-10-22 09:19:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1702 [2026-10-22 09:19:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4306 [2026-10-22 09:19:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4451 [2026-10-22 09:19:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8496 [2026-10-22 09:19:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0949 [2026-10-22 09:19:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4010 [2026-10-22 09:19:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3874 [2026-10-22 09:19:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2424 [2026-10-22 09:19:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5532 [2026-10-22 09:20:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8576 [2026-10-22 09:20:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9850 [2026-10-22 09:20:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7885 [2026-10-22 09:20:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8199 [2026-10-22 09:20:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1497 [2026-10-22 09:20:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2838 [2026-10-22 09:20:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7399 [2026-10-22 09:20:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6675 [2026-10-22 09:20:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8850 [2026-10-22 09:20:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7282 [2026-10-22 09:20:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8354 [2026-10-22 09:20:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3373 [2026-10-22 09:21:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4979 [2026-10-22 09:21:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.1828 [2026-10-22 09:21:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5280 [2026-10-22 09:21:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6386 [2026-10-22 09:21:13][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,607 | MFU: 1.8% [2026-10-22 09:21:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4012 [2026-10-22 09:21:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4027 [2026-10-22 09:21:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5249 [2026-10-22 09:21:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.1327 [2026-10-22 09:21:19][INFO][training.pretrain_engine] [Ph2|Step 666] loss=4.6314 | ppl=102.66 | aux=0.02506 | lr=2.97e-04 | gnorm=0.520 | tok/s=29,242 | MFU=7.0% | step=143432ms | tokens=1.501B | mem=4.2/32.7GB | cost=$54.65/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 17:35 (128.2h) | Phase3: SKIPPED | SFT: Oct 27 22:20 (133.0h) | DPO (Final): Oct 27 22:54 (133.6h) [2026-10-22 09:21:22][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step666 | Tokens: 1.501B | Spent: $54.65 | Remaining: $945.35 [2026-10-22 09:21:22][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step666 [2026-10-22 09:21:22][INFO][training.pretrain_engine] Starting global step 667 (accumulating 32 micro-batches)... [2026-10-22 09:21:22][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 09:21:22][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-22 09:21:22][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-22 09:21:22][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-22 09:21:22][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step666 to Phase-Technologies/forge-500m-pretrain (path: phase2_step666)... [2026-10-22 09:21:22][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-22 09:21:32][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-22 09:21:32][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step666 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-22 09:21:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3919 [2026-10-22 09:21:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7801 [2026-10-22 09:21:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5351 [2026-10-22 09:21:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1404 [2026-10-22 09:21:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4263 [2026-10-22 09:21:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4194 [2026-10-22 09:21:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5668 [2026-10-22 09:21:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1150 [2026-10-22 09:22:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0379 [2026-10-22 09:22:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5805 [2026-10-22 09:22:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8701 [2026-10-22 09:22:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7257 [2026-10-22 09:22:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3173 [2026-10-22 09:22:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6620 [2026-10-22 09:22:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6989 [2026-10-22 09:22:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3462 [2026-10-22 09:22:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7440 [2026-10-22 09:22:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0966 [2026-10-22 09:22:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4183 [2026-10-22 09:22:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7032 [2026-10-22 09:23:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.1070 [2026-10-22 09:23:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5037 [2026-10-22 09:23:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7145 [2026-10-22 09:23:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.0013 [2026-10-22 09:23:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0284 [2026-10-22 09:23:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7081 [2026-10-22 09:23:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1400 [2026-10-22 09:23:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9450 [2026-10-22 09:23:40][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,612 | MFU: 1.8% [2026-10-22 09:23:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9798 [2026-10-22 09:23:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5415 [2026-10-22 09:23:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9088 [2026-10-22 09:23:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8639 [2026-10-22 09:23:44][INFO][training.pretrain_engine] [Ph2|Step 667] loss=4.8570 | ppl=128.64 | aux=0.02505 | lr=2.97e-04 | gnorm=0.535 | tok/s=29,245 | MFU=7.0% | step=143420ms | tokens=1.502B | mem=4.2/32.7GB | cost=$54.73/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 17:36 (128.2h) | Phase3: SKIPPED | SFT: Oct 27 22:21 (133.0h) | DPO (Final): Oct 27 22:55 (133.5h) [2026-10-22 09:23:44][INFO][training.pretrain_engine] Starting global step 668 (accumulating 32 micro-batches)... [2026-10-22 09:23:44][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 09:24:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4455 [2026-10-22 09:24:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6284 [2026-10-22 09:24:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5687 [2026-10-22 09:24:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9267 [2026-10-22 09:24:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.9768 [2026-10-22 09:24:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5570 [2026-10-22 09:24:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.0473 [2026-10-22 09:24:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3786 [2026-10-22 09:24:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7866 [2026-10-22 09:24:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8785 [2026-10-22 09:24:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9714 [2026-10-22 09:24:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9258 [2026-10-22 09:24:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9289 [2026-10-22 09:24:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4503 [2026-10-22 09:24:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3594 [2026-10-22 09:24:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3256 [2026-10-22 09:25:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2245 [2026-10-22 09:25:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7891 [2026-10-22 09:25:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7980 [2026-10-22 09:25:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5250 [2026-10-22 09:25:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6636 [2026-10-22 09:25:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6749 [2026-10-22 09:25:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1562 [2026-10-22 09:25:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6257 [2026-10-22 09:25:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1662 [2026-10-22 09:25:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5878 [2026-10-22 09:25:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6483 [2026-10-22 09:25:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1804 [2026-10-22 09:26:02][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,599 | MFU: 1.8% [2026-10-22 09:26:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5595 [2026-10-22 09:26:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.0151 [2026-10-22 09:26:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8713 [2026-10-22 09:26:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.1556 [2026-10-22 09:26:08][INFO][training.pretrain_engine] [Ph2|Step 668] loss=4.8706 | ppl=130.40 | aux=0.02505 | lr=2.97e-04 | gnorm=0.496 | tok/s=29,244 | MFU=7.0% | step=143425ms | tokens=1.503B | mem=4.2/32.7GB | cost=$54.82/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 17:38 (128.2h) | Phase3: SKIPPED | SFT: Oct 27 22:23 (133.0h) | DPO (Final): Oct 27 22:57 (133.5h) [2026-10-22 09:26:08][INFO][training.pretrain_engine] Starting global step 669 (accumulating 32 micro-batches)... [2026-10-22 09:26:08][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 09:26:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8493 [2026-10-22 09:26:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5383 [2026-10-22 09:26:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1422 [2026-10-22 09:26:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.7910 [2026-10-22 09:26:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0235 [2026-10-22 09:26:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2588 [2026-10-22 09:26:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6320 [2026-10-22 09:26:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7708 [2026-10-22 09:27:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2357 [2026-10-22 09:27:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0512 [2026-10-22 09:27:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4269 [2026-10-22 09:27:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3017 [2026-10-22 09:27:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9188 [2026-10-22 09:27:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4675 [2026-10-22 09:27:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4580 [2026-10-22 09:27:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4523 [2026-10-22 09:27:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5761 [2026-10-22 09:27:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=3.9578 [2026-10-22 09:27:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7034 [2026-10-22 09:27:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9468 [2026-10-22 09:27:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9863 [2026-10-22 09:27:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5866 [2026-10-22 09:27:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7346 [2026-10-22 09:27:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2301 [2026-10-22 09:28:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2328 [2026-10-22 09:28:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6457 [2026-10-22 09:28:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2403 [2026-10-22 09:28:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2803 [2026-10-22 09:28:26][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,599 | MFU: 1.8% [2026-10-22 09:28:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3645 [2026-10-22 09:28:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6297 [2026-10-22 09:28:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7494 [2026-10-22 09:28:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9113 [2026-10-22 09:28:30][INFO][training.pretrain_engine] [Ph2|Step 669] loss=4.7568 | ppl=116.38 | aux=0.02506 | lr=2.97e-04 | gnorm=0.494 | tok/s=29,246 | MFU=7.0% | step=143417ms | tokens=1.504B | mem=4.2/32.7GB | cost=$54.90/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 17:39 (128.2h) | Phase3: SKIPPED | SFT: Oct 27 22:24 (132.9h) | DPO (Final): Oct 27 22:59 (133.5h) [2026-10-22 09:28:30][INFO][training.pretrain_engine] Starting global step 670 (accumulating 32 micro-batches)... [2026-10-22 09:28:30][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 09:28:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5352 [2026-10-22 09:28:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0543 [2026-10-22 09:28:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0863 [2026-10-22 09:28:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7928 [2026-10-22 09:29:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8209 [2026-10-22 09:29:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4226 [2026-10-22 09:29:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3728 [2026-10-22 09:29:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6834 [2026-10-22 09:29:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4769 [2026-10-22 09:29:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8246 [2026-10-22 09:29:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1347 [2026-10-22 09:29:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6912 [2026-10-22 09:29:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4797 [2026-10-22 09:29:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0600 [2026-10-22 09:29:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8730 [2026-10-22 09:29:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2434 [2026-10-22 09:29:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8290 [2026-10-22 09:29:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1151 [2026-10-22 09:29:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0483 [2026-10-22 09:29:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9670 [2026-10-22 09:30:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9050 [2026-10-22 09:30:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1591 [2026-10-22 09:30:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6531 [2026-10-22 09:30:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7840 [2026-10-22 09:30:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6021 [2026-10-22 09:30:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1633 [2026-10-22 09:30:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2764 [2026-10-22 09:30:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9488 [2026-10-22 09:30:48][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,598 | MFU: 1.8% [2026-10-22 09:30:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6855 [2026-10-22 09:30:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9002 [2026-10-22 09:30:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4852 [2026-10-22 09:30:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5732 [2026-10-22 09:30:53][INFO][training.pretrain_engine] [Ph2|Step 670] loss=4.7264 | ppl=112.89 | aux=0.02505 | lr=2.97e-04 | gnorm=0.535 | tok/s=29,246 | MFU=7.0% | step=143415ms | tokens=1.505B | mem=4.2/32.7GB | cost=$54.98/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 17:41 (128.2h) | Phase3: SKIPPED | SFT: Oct 27 22:26 (132.9h) | DPO (Final): Oct 27 23:00 (133.5h) [2026-10-22 09:30:53][INFO][training.pretrain_engine] Starting global step 671 (accumulating 32 micro-batches)... [2026-10-22 09:30:53][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 09:31:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6923 [2026-10-22 09:31:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9010 [2026-10-22 09:31:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7849 [2026-10-22 09:31:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0724 [2026-10-22 09:31:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1228 [2026-10-22 09:31:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1460 [2026-10-22 09:31:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4413 [2026-10-22 09:31:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4524 [2026-10-22 09:31:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1496 [2026-10-22 09:31:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5282 [2026-10-22 09:31:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5025 [2026-10-22 09:31:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3491 [2026-10-22 09:32:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7274 [2026-10-22 09:32:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3150 [2026-10-22 09:32:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7569 [2026-10-22 09:32:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9796 [2026-10-22 09:32:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8351 [2026-10-22 09:32:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7307 [2026-10-22 09:32:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4845 [2026-10-22 09:32:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7370 [2026-10-22 09:32:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9680 [2026-10-22 09:32:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4995 [2026-10-22 09:32:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5483 [2026-10-22 09:32:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0664 [2026-10-22 09:32:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4076 [2026-10-22 09:32:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1991 [2026-10-22 09:32:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4769 [2026-10-22 09:32:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3198 [2026-10-22 09:33:11][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,592 | MFU: 1.8% [2026-10-22 09:33:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8001 [2026-10-22 09:33:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7447 [2026-10-22 09:33:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7827 [2026-10-22 09:33:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6183 [2026-10-22 09:33:16][INFO][training.pretrain_engine] [Ph2|Step 671] loss=4.8238 | ppl=124.43 | aux=0.02505 | lr=2.97e-04 | gnorm=0.617 | tok/s=29,246 | MFU=7.0% | step=143413ms | tokens=1.506B | mem=4.2/32.7GB | cost=$55.06/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 17:43 (128.2h) | Phase3: SKIPPED | SFT: Oct 27 22:28 (132.9h) | DPO (Final): Oct 27 23:02 (133.5h) [2026-10-22 09:33:16][INFO][training.pretrain_engine] Starting global step 672 (accumulating 32 micro-batches)... [2026-10-22 09:33:16][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 09:33:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3172 [2026-10-22 09:33:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9462 [2026-10-22 09:33:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7737 [2026-10-22 09:33:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1391 [2026-10-22 09:33:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7961 [2026-10-22 09:33:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3405 [2026-10-22 09:33:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2699 [2026-10-22 09:33:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1954 [2026-10-22 09:34:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.0434 [2026-10-22 09:34:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.1287 [2026-10-22 09:34:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2482 [2026-10-22 09:34:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3977 [2026-10-22 09:34:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9248 [2026-10-22 09:34:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6985 [2026-10-22 09:34:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7799 [2026-10-22 09:34:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8241 [2026-10-22 09:34:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2164 [2026-10-22 09:34:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9901 [2026-10-22 09:34:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0481 [2026-10-22 09:34:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3552 [2026-10-22 09:35:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9114 [2026-10-22 09:35:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7177 [2026-10-22 09:35:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2144 [2026-10-22 09:35:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7674 [2026-10-22 09:35:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7508 [2026-10-22 09:35:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4126 [2026-10-22 09:35:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2391 [2026-10-22 09:35:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8538 [2026-10-22 09:35:34][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,597 | MFU: 1.8% [2026-10-22 09:35:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=3.9142 [2026-10-22 09:35:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6629 [2026-10-22 09:35:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6804 [2026-10-22 09:35:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8627 [2026-10-22 09:35:39][INFO][training.pretrain_engine] [Ph2|Step 672] loss=4.8046 | ppl=122.07 | aux=0.02505 | lr=2.97e-04 | gnorm=0.668 | tok/s=29,248 | MFU=7.0% | step=143405ms | tokens=1.507B | mem=4.2/32.7GB | cost=$55.14/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 17:44 (128.1h) | Phase3: SKIPPED | SFT: Oct 27 22:29 (132.9h) | DPO (Final): Oct 27 23:03 (133.5h) [2026-10-22 09:35:39][INFO][training.pretrain_engine] Starting global step 673 (accumulating 32 micro-batches)... [2026-10-22 09:35:39][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 09:35:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7157 [2026-10-22 09:35:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6799 [2026-10-22 09:35:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4207 [2026-10-22 09:35:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3696 [2026-10-22 09:36:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2180 [2026-10-22 09:36:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4127 [2026-10-22 09:36:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5580 [2026-10-22 09:36:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8158 [2026-10-22 09:36:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8674 [2026-10-22 09:36:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0426 [2026-10-22 09:36:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5179 [2026-10-22 09:36:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6940 [2026-10-22 09:36:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1703 [2026-10-22 09:36:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2746 [2026-10-22 09:36:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7837 [2026-10-22 09:36:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0488 [2026-10-22 09:37:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8375 [2026-10-22 09:37:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5113 [2026-10-22 09:37:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4275 [2026-10-22 09:37:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7904 [2026-10-22 09:37:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0328 [2026-10-22 09:37:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7198 [2026-10-22 09:37:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1509 [2026-10-22 09:37:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6705 [2026-10-22 09:37:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8677 [2026-10-22 09:37:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6502 [2026-10-22 09:37:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6204 [2026-10-22 09:37:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0245 [2026-10-22 09:37:57][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,605 | MFU: 1.8% [2026-10-22 09:38:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0454 [2026-10-22 09:38:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8713 [2026-10-22 09:38:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1832 [2026-10-22 09:38:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8051 [2026-10-22 09:38:02][INFO][training.pretrain_engine] [Ph2|Step 673] loss=4.9242 | ppl=137.57 | aux=0.02505 | lr=2.97e-04 | gnorm=0.730 | tok/s=29,248 | MFU=7.0% | step=143403ms | tokens=1.508B | mem=4.2/32.7GB | cost=$55.23/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 17:46 (128.1h) | Phase3: SKIPPED | SFT: Oct 27 22:31 (132.9h) | DPO (Final): Oct 27 23:05 (133.5h) [2026-10-22 09:38:02][INFO][training.pretrain_engine] Starting global step 674 (accumulating 32 micro-batches)... [2026-10-22 09:38:02][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 09:38:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0010 [2026-10-22 09:38:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0073 [2026-10-22 09:38:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7213 [2026-10-22 09:38:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8394 [2026-10-22 09:38:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0103 [2026-10-22 09:38:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7528 [2026-10-22 09:38:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5387 [2026-10-22 09:38:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7373 [2026-10-22 09:38:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5041 [2026-10-22 09:38:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5577 [2026-10-22 09:38:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5397 [2026-10-22 09:38:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.0527 [2026-10-22 09:39:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7825 [2026-10-22 09:39:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9677 [2026-10-22 09:39:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8829 [2026-10-22 09:39:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7348 [2026-10-22 09:39:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5903 [2026-10-22 09:39:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7205 [2026-10-22 09:39:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9265 [2026-10-22 09:39:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6618 [2026-10-22 09:39:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0218 [2026-10-22 09:39:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8306 [2026-10-22 09:39:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1302 [2026-10-22 09:39:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1633 [2026-10-22 09:40:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8714 [2026-10-22 09:40:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8691 [2026-10-22 09:40:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0004 [2026-10-22 09:40:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2259 [2026-10-22 09:40:20][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,614 | MFU: 1.8% [2026-10-22 09:40:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5103 [2026-10-22 09:40:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2251 [2026-10-22 09:40:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9715 [2026-10-22 09:40:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5258 [2026-10-22 09:40:25][INFO][training.pretrain_engine] [Ph2|Step 674] loss=4.8234 | ppl=124.39 | aux=0.02505 | lr=2.97e-04 | gnorm=0.938 | tok/s=29,250 | MFU=7.0% | step=143393ms | tokens=1.509B | mem=4.2/32.7GB | cost=$55.31/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 17:47 (128.1h) | Phase3: SKIPPED | SFT: Oct 27 22:32 (132.9h) | DPO (Final): Oct 27 23:06 (133.4h) [2026-10-22 09:40:25][INFO][training.pretrain_engine] Starting global step 675 (accumulating 32 micro-batches)... [2026-10-22 09:40:25][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 09:40:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6007 [2026-10-22 09:40:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6789 [2026-10-22 09:40:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5854 [2026-10-22 09:40:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9360 [2026-10-22 09:41:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3341 [2026-10-22 09:41:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4488 [2026-10-22 09:41:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3647 [2026-10-22 09:41:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1611 [2026-10-22 09:41:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2291 [2026-10-22 09:41:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8886 [2026-10-22 09:41:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0086 [2026-10-22 09:41:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0058 [2026-10-22 09:41:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9799 [2026-10-22 09:41:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9959 [2026-10-22 09:41:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7110 [2026-10-22 09:41:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8676 [2026-10-22 09:41:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3170 [2026-10-22 09:41:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.8874 [2026-10-22 09:41:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1235 [2026-10-22 09:41:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3041 [2026-10-22 09:42:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8691 [2026-10-22 09:42:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6921 [2026-10-22 09:42:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4918 [2026-10-22 09:42:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4778 [2026-10-22 09:42:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5594 [2026-10-22 09:42:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2946 [2026-10-22 09:42:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9115 [2026-10-22 09:42:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5870 [2026-10-22 09:42:43][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,603 | MFU: 1.8% [2026-10-22 09:42:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9299 [2026-10-22 09:42:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9970 [2026-10-22 09:42:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9419 [2026-10-22 09:42:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9846 [2026-10-22 09:42:48][INFO][training.pretrain_engine] [Ph2|Step 675] loss=4.8907 | ppl=133.05 | aux=0.02507 | lr=2.97e-04 | gnorm=1.039 | tok/s=29,252 | MFU=7.0% | step=143386ms | tokens=1.510B | mem=4.2/32.7GB | cost=$55.39/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 17:48 (128.1h) | Phase3: SKIPPED | SFT: Oct 27 22:33 (132.9h) | DPO (Final): Oct 27 23:08 (133.4h) [2026-10-22 09:42:51][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step675 | Tokens: 1.510B | Spent: $55.39 | Remaining: $944.61 [2026-10-22 09:42:51][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step675 [2026-10-22 09:42:51][INFO][training.pretrain_engine] Starting global step 676 (accumulating 32 micro-batches)... [2026-10-22 09:42:51][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 09:42:51][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-22 09:42:51][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-22 09:42:51][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-22 09:42:51][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step675 to Phase-Technologies/forge-500m-pretrain (path: phase2_step675)... [2026-10-22 09:42:52][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-22 09:43:01][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-22 09:43:01][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step675 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-22 09:43:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5468 [2026-10-22 09:43:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6605 [2026-10-22 09:43:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8239 [2026-10-22 09:43:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1665 [2026-10-22 09:43:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7014 [2026-10-22 09:43:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6798 [2026-10-22 09:43:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6259 [2026-10-22 09:43:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9207 [2026-10-22 09:43:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.0532 [2026-10-22 09:43:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2877 [2026-10-22 09:43:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2977 [2026-10-22 09:43:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4206 [2026-10-22 09:43:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4753 [2026-10-22 09:43:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5691 [2026-10-22 09:44:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2043 [2026-10-22 09:44:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3457 [2026-10-22 09:44:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5174 [2026-10-22 09:44:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2293 [2026-10-22 09:44:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9387 [2026-10-22 09:44:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9650 [2026-10-22 09:44:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8864 [2026-10-22 09:44:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3500 [2026-10-22 09:44:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6400 [2026-10-22 09:44:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2669 [2026-10-22 09:44:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9719 [2026-10-22 09:44:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9964 [2026-10-22 09:44:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2559 [2026-10-22 09:44:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7613 [2026-10-22 09:45:09][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,589 | MFU: 1.8% [2026-10-22 09:45:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7147 [2026-10-22 09:45:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6836 [2026-10-22 09:45:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0960 [2026-10-22 09:45:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1758 [2026-10-22 09:45:14][INFO][training.pretrain_engine] [Ph2|Step 676] loss=4.7930 | ppl=120.66 | aux=0.02505 | lr=2.97e-04 | gnorm=0.836 | tok/s=29,254 | MFU=7.0% | step=143376ms | tokens=1.511B | mem=4.2/32.7GB | cost=$55.47/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 17:50 (128.1h) | Phase3: SKIPPED | SFT: Oct 27 22:35 (132.8h) | DPO (Final): Oct 27 23:09 (133.4h) [2026-10-22 09:45:14][INFO][training.pretrain_engine] Starting global step 677 (accumulating 32 micro-batches)... [2026-10-22 09:45:14][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 09:45:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.1076 [2026-10-22 09:45:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.8268 [2026-10-22 09:45:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3127 [2026-10-22 09:45:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3543 [2026-10-22 09:45:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8116 [2026-10-22 09:45:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9770 [2026-10-22 09:45:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1034 [2026-10-22 09:45:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7469 [2026-10-22 09:46:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0042 [2026-10-22 09:46:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4088 [2026-10-22 09:46:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2777 [2026-10-22 09:46:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0835 [2026-10-22 09:46:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7314 [2026-10-22 09:46:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9819 [2026-10-22 09:46:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0625 [2026-10-22 09:46:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6669 [2026-10-22 09:46:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8670 [2026-10-22 09:46:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6536 [2026-10-22 09:46:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1033 [2026-10-22 09:46:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=3.7279 [2026-10-22 09:47:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8711 [2026-10-22 09:47:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9953 [2026-10-22 09:47:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5704 [2026-10-22 09:47:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8773 [2026-10-22 09:47:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5675 [2026-10-22 09:47:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6714 [2026-10-22 09:47:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8547 [2026-10-22 09:47:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7303 [2026-10-22 09:47:32][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,575 | MFU: 1.8% [2026-10-22 09:47:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5889 [2026-10-22 09:47:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7158 [2026-10-22 09:47:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6385 [2026-10-22 09:47:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1502 [2026-10-22 09:47:37][INFO][training.pretrain_engine] [Ph2|Step 677] loss=4.8168 | ppl=123.57 | aux=0.02506 | lr=2.97e-04 | gnorm=0.926 | tok/s=29,254 | MFU=7.0% | step=143377ms | tokens=1.512B | mem=4.2/32.7GB | cost=$55.55/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 17:52 (128.1h) | Phase3: SKIPPED | SFT: Oct 27 22:36 (132.8h) | DPO (Final): Oct 27 23:11 (133.4h) [2026-10-22 09:47:37][INFO][training.pretrain_engine] Starting global step 678 (accumulating 32 micro-batches)... [2026-10-22 09:47:37][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 09:47:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6172 [2026-10-22 09:47:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2179 [2026-10-22 09:47:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7489 [2026-10-22 09:47:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.9485 [2026-10-22 09:48:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9984 [2026-10-22 09:48:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3371 [2026-10-22 09:48:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9500 [2026-10-22 09:48:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3233 [2026-10-22 09:48:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3086 [2026-10-22 09:48:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6819 [2026-10-22 09:48:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6364 [2026-10-22 09:48:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3495 [2026-10-22 09:48:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9121 [2026-10-22 09:48:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7384 [2026-10-22 09:48:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8666 [2026-10-22 09:48:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1469 [2026-10-22 09:49:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8682 [2026-10-22 09:49:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7122 [2026-10-22 09:49:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8759 [2026-10-22 09:49:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6723 [2026-10-22 09:49:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.1836 [2026-10-22 09:49:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5057 [2026-10-22 09:49:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0157 [2026-10-22 09:49:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6114 [2026-10-22 09:49:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0321 [2026-10-22 09:49:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3433 [2026-10-22 09:49:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8670 [2026-10-22 09:49:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4002 [2026-10-22 09:49:55][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,605 | MFU: 1.8% [2026-10-22 09:50:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8120 [2026-10-22 09:50:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5306 [2026-10-22 09:50:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5172 [2026-10-22 09:50:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6439 [2026-10-22 09:50:00][INFO][training.pretrain_engine] [Ph2|Step 678] loss=4.7626 | ppl=117.05 | aux=0.02506 | lr=2.97e-04 | gnorm=0.840 | tok/s=29,255 | MFU=7.0% | step=143373ms | tokens=1.513B | mem=4.2/32.7GB | cost=$55.64/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 17:53 (128.1h) | Phase3: SKIPPED | SFT: Oct 27 22:38 (132.8h) | DPO (Final): Oct 27 23:12 (133.4h) [2026-10-22 09:50:00][INFO][training.pretrain_engine] Starting global step 679 (accumulating 32 micro-batches)... [2026-10-22 09:50:00][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 09:50:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0521 [2026-10-22 09:50:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1341 [2026-10-22 09:50:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9225 [2026-10-22 09:50:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7775 [2026-10-22 09:50:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8965 [2026-10-22 09:50:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8972 [2026-10-22 09:50:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7444 [2026-10-22 09:50:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5801 [2026-10-22 09:50:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0915 [2026-10-22 09:50:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0447 [2026-10-22 09:50:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2705 [2026-10-22 09:50:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2470 [2026-10-22 09:51:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7879 [2026-10-22 09:51:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0263 [2026-10-22 09:51:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5828 [2026-10-22 09:51:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7333 [2026-10-22 09:51:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9857 [2026-10-22 09:51:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0400 [2026-10-22 09:51:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7767 [2026-10-22 09:51:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6655 [2026-10-22 09:51:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2927 [2026-10-22 09:51:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7644 [2026-10-22 09:51:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0944 [2026-10-22 09:51:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7247 [2026-10-22 09:52:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8011 [2026-10-22 09:52:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5365 [2026-10-22 09:52:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8016 [2026-10-22 09:52:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4893 [2026-10-22 09:52:18][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,593 | MFU: 1.8% [2026-10-22 09:52:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2906 [2026-10-22 09:52:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8342 [2026-10-22 09:52:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6950 [2026-10-22 09:52:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1875 [2026-10-22 09:52:23][INFO][training.pretrain_engine] [Ph2|Step 679] loss=4.7737 | ppl=118.35 | aux=0.02505 | lr=2.97e-04 | gnorm=0.793 | tok/s=29,256 | MFU=7.0% | step=143367ms | tokens=1.514B | mem=4.2/32.7GB | cost=$55.72/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 17:55 (128.0h) | Phase3: SKIPPED | SFT: Oct 27 22:39 (132.8h) | DPO (Final): Oct 27 23:14 (133.4h) [2026-10-22 09:52:23][INFO][training.pretrain_engine] Starting global step 680 (accumulating 32 micro-batches)... [2026-10-22 09:52:23][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 09:52:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0046 [2026-10-22 09:52:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9218 [2026-10-22 09:52:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9383 [2026-10-22 09:52:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1707 [2026-10-22 09:52:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2684 [2026-10-22 09:52:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4553 [2026-10-22 09:52:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4163 [2026-10-22 09:52:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5337 [2026-10-22 09:53:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7042 [2026-10-22 09:53:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4730 [2026-10-22 09:53:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1052 [2026-10-22 09:53:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0225 [2026-10-22 09:53:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0843 [2026-10-22 09:53:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5238 [2026-10-22 09:53:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6300 [2026-10-22 09:53:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9865 [2026-10-22 09:53:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2062 [2026-10-22 09:53:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5412 [2026-10-22 09:53:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9760 [2026-10-22 09:53:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3430 [2026-10-22 09:54:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7034 [2026-10-22 09:54:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1292 [2026-10-22 09:54:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9247 [2026-10-22 09:54:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2153 [2026-10-22 09:54:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8559 [2026-10-22 09:54:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9788 [2026-10-22 09:54:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.0077 [2026-10-22 09:54:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4994 [2026-10-22 09:54:40][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,610 | MFU: 1.8% [2026-10-22 09:54:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1806 [2026-10-22 09:54:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7442 [2026-10-22 09:54:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9065 [2026-10-22 09:54:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1158 [2026-10-22 09:54:46][INFO][training.pretrain_engine] [Ph2|Step 680] loss=4.8637 | ppl=129.50 | aux=0.02505 | lr=2.97e-04 | gnorm=0.762 | tok/s=29,256 | MFU=7.0% | step=143365ms | tokens=1.515B | mem=4.2/32.7GB | cost=$55.80/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 17:56 (128.0h) | Phase3: SKIPPED | SFT: Oct 27 22:41 (132.8h) | DPO (Final): Oct 27 23:15 (133.4h) [2026-10-22 09:54:46][INFO][training.pretrain_engine] Starting global step 681 (accumulating 32 micro-batches)... [2026-10-22 09:54:46][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 09:55:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6777 [2026-10-22 09:55:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6868 [2026-10-22 09:55:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5604 [2026-10-22 09:55:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4666 [2026-10-22 09:55:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6000 [2026-10-22 09:55:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0404 [2026-10-22 09:55:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1799 [2026-10-22 09:55:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3224 [2026-10-22 09:55:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9181 [2026-10-22 09:55:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9507 [2026-10-22 09:55:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7940 [2026-10-22 09:55:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8116 [2026-10-22 09:55:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8034 [2026-10-22 09:55:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7792 [2026-10-22 09:55:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7919 [2026-10-22 09:55:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1861 [2026-10-22 09:56:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.2805 [2026-10-22 09:56:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5426 [2026-10-22 09:56:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8602 [2026-10-22 09:56:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6478 [2026-10-22 09:56:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1612 [2026-10-22 09:56:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2010 [2026-10-22 09:56:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9650 [2026-10-22 09:56:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6039 [2026-10-22 09:56:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9884 [2026-10-22 09:56:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2807 [2026-10-22 09:56:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6462 [2026-10-22 09:56:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7813 [2026-10-22 09:57:03][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,604 | MFU: 1.8% [2026-10-22 09:57:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8807 [2026-10-22 09:57:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4396 [2026-10-22 09:57:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9247 [2026-10-22 09:57:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9858 [2026-10-22 09:57:08][INFO][training.pretrain_engine] [Ph2|Step 681] loss=4.8403 | ppl=126.51 | aux=0.02506 | lr=2.97e-04 | gnorm=0.758 | tok/s=29,258 | MFU=7.0% | step=143356ms | tokens=1.516B | mem=4.2/32.7GB | cost=$55.88/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 17:58 (128.0h) | Phase3: SKIPPED | SFT: Oct 27 22:42 (132.8h) | DPO (Final): Oct 27 23:17 (133.3h) [2026-10-22 09:57:08][INFO][training.pretrain_engine] Starting global step 682 (accumulating 32 micro-batches)... [2026-10-22 09:57:08][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 09:57:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.0665 [2026-10-22 09:57:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2750 [2026-10-22 09:57:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8236 [2026-10-22 09:57:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5316 [2026-10-22 09:57:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.0473 [2026-10-22 09:57:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4096 [2026-10-22 09:57:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.9925 [2026-10-22 09:57:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2734 [2026-10-22 09:58:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6733 [2026-10-22 09:58:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0854 [2026-10-22 09:58:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9467 [2026-10-22 09:58:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0000 [2026-10-22 09:58:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8624 [2026-10-22 09:58:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9355 [2026-10-22 09:58:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0626 [2026-10-22 09:58:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3735 [2026-10-22 09:58:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=3.9971 [2026-10-22 09:58:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7216 [2026-10-22 09:58:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1251 [2026-10-22 09:58:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2385 [2026-10-22 09:58:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9874 [2026-10-22 09:58:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0472 [2026-10-22 09:58:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6092 [2026-10-22 09:58:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1687 [2026-10-22 09:59:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8385 [2026-10-22 09:59:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9043 [2026-10-22 09:59:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9326 [2026-10-22 09:59:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9194 [2026-10-22 09:59:26][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,622 | MFU: 1.8% [2026-10-22 09:59:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6892 [2026-10-22 09:59:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1404 [2026-10-22 09:59:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8986 [2026-10-22 09:59:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7289 [2026-10-22 09:59:31][INFO][training.pretrain_engine] [Ph2|Step 682] loss=4.6658 | ppl=106.25 | aux=0.02506 | lr=2.97e-04 | gnorm=0.645 | tok/s=29,262 | MFU=7.0% | step=143337ms | tokens=1.517B | mem=4.2/32.7GB | cost=$55.96/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 17:58 (128.0h) | Phase3: SKIPPED | SFT: Oct 27 22:43 (132.7h) | DPO (Final): Oct 27 23:17 (133.3h) [2026-10-22 09:59:31][INFO][training.pretrain_engine] Starting global step 683 (accumulating 32 micro-batches)... [2026-10-22 09:59:31][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 09:59:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7959 [2026-10-22 09:59:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8317 [2026-10-22 09:59:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8543 [2026-10-22 09:59:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7305 [2026-10-22 10:00:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5612 [2026-10-22 10:00:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4774 [2026-10-22 10:00:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4721 [2026-10-22 10:00:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9112 [2026-10-22 10:00:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0637 [2026-10-22 10:00:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7099 [2026-10-22 10:00:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7386 [2026-10-22 10:00:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7480 [2026-10-22 10:00:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3925 [2026-10-22 10:00:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2989 [2026-10-22 10:00:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7174 [2026-10-22 10:00:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.1228 [2026-10-22 10:01:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8489 [2026-10-22 10:01:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6857 [2026-10-22 10:01:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5710 [2026-10-22 10:01:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9181 [2026-10-22 10:01:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8097 [2026-10-22 10:01:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2564 [2026-10-22 10:01:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3904 [2026-10-22 10:01:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8560 [2026-10-22 10:01:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9260 [2026-10-22 10:01:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2546 [2026-10-22 10:01:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9311 [2026-10-22 10:01:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7277 [2026-10-22 10:01:49][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,613 | MFU: 1.8% [2026-10-22 10:01:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6958 [2026-10-22 10:01:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6794 [2026-10-22 10:01:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1735 [2026-10-22 10:01:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6909 [2026-10-22 10:01:54][INFO][training.pretrain_engine] [Ph2|Step 683] loss=4.7615 | ppl=116.93 | aux=0.02506 | lr=2.97e-04 | gnorm=0.625 | tok/s=29,263 | MFU=7.0% | step=143333ms | tokens=1.518B | mem=4.2/32.7GB | cost=$56.05/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 18:00 (128.0h) | Phase3: SKIPPED | SFT: Oct 27 22:45 (132.7h) | DPO (Final): Oct 27 23:19 (133.3h) [2026-10-22 10:01:54][INFO][training.pretrain_engine] Starting global step 684 (accumulating 32 micro-batches)... [2026-10-22 10:01:54][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 10:02:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0337 [2026-10-22 10:02:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9626 [2026-10-22 10:02:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1819 [2026-10-22 10:02:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0922 [2026-10-22 10:02:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8478 [2026-10-22 10:02:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3719 [2026-10-22 10:02:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8067 [2026-10-22 10:02:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5852 [2026-10-22 10:02:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9599 [2026-10-22 10:02:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2468 [2026-10-22 10:02:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2960 [2026-10-22 10:02:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9417 [2026-10-22 10:03:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=3.9458 [2026-10-22 10:03:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4803 [2026-10-22 10:03:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8622 [2026-10-22 10:03:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3984 [2026-10-22 10:03:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6209 [2026-10-22 10:03:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5412 [2026-10-22 10:03:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0177 [2026-10-22 10:03:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7629 [2026-10-22 10:03:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9757 [2026-10-22 10:03:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8476 [2026-10-22 10:03:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0636 [2026-10-22 10:03:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1080 [2026-10-22 10:03:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4200 [2026-10-22 10:03:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4301 [2026-10-22 10:03:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8426 [2026-10-22 10:03:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7948 [2026-10-22 10:04:12][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,621 | MFU: 1.8% [2026-10-22 10:04:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=3.8994 [2026-10-22 10:04:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5377 [2026-10-22 10:04:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5113 [2026-10-22 10:04:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7739 [2026-10-22 10:04:17][INFO][training.pretrain_engine] [Ph2|Step 684] loss=4.7799 | ppl=119.09 | aux=0.02506 | lr=2.97e-04 | gnorm=0.660 | tok/s=29,264 | MFU=7.0% | step=143329ms | tokens=1.519B | mem=4.2/32.7GB | cost=$56.13/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 18:01 (128.0h) | Phase3: SKIPPED | SFT: Oct 27 22:46 (132.7h) | DPO (Final): Oct 27 23:20 (133.3h) [2026-10-22 10:04:20][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step684 | Tokens: 1.519B | Spent: $56.13 | Remaining: $943.87 [2026-10-22 10:04:20][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step684 [2026-10-22 10:04:20][INFO][training.pretrain_engine] Starting global step 685 (accumulating 32 micro-batches)... [2026-10-22 10:04:20][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 10:04:20][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-22 10:04:20][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-22 10:04:20][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-22 10:04:20][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step684 to Phase-Technologies/forge-500m-pretrain (path: phase2_step684)... [2026-10-22 10:04:20][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-22 10:04:30][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-22 10:04:30][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step684 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-22 10:04:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.1985 [2026-10-22 10:04:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.5211 [2026-10-22 10:04:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.7593 [2026-10-22 10:04:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.1583 [2026-10-22 10:04:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4936 [2026-10-22 10:04:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5458 [2026-10-22 10:04:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.0482 [2026-10-22 10:04:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4192 [2026-10-22 10:05:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5192 [2026-10-22 10:05:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7239 [2026-10-22 10:05:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6350 [2026-10-22 10:05:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6788 [2026-10-22 10:05:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9688 [2026-10-22 10:05:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0526 [2026-10-22 10:05:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2608 [2026-10-22 10:05:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0170 [2026-10-22 10:05:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0996 [2026-10-22 10:05:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4768 [2026-10-22 10:05:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6802 [2026-10-22 10:05:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5539 [2026-10-22 10:06:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3932 [2026-10-22 10:06:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6869 [2026-10-22 10:06:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9436 [2026-10-22 10:06:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8129 [2026-10-22 10:06:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6570 [2026-10-22 10:06:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4304 [2026-10-22 10:06:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8704 [2026-10-22 10:06:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6728 [2026-10-22 10:06:38][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,617 | MFU: 1.8% [2026-10-22 10:06:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1598 [2026-10-22 10:06:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9721 [2026-10-22 10:06:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1019 [2026-10-22 10:06:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9954 [2026-10-22 10:06:42][INFO][training.pretrain_engine] [Ph2|Step 685] loss=4.7382 | ppl=114.23 | aux=0.02506 | lr=2.97e-04 | gnorm=0.555 | tok/s=29,268 | MFU=7.0% | step=143307ms | tokens=1.520B | mem=4.2/32.7GB | cost=$56.21/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 18:02 (127.9h) | Phase3: SKIPPED | SFT: Oct 27 22:47 (132.7h) | DPO (Final): Oct 27 23:21 (133.2h) [2026-10-22 10:06:42][INFO][training.pretrain_engine] Starting global step 686 (accumulating 32 micro-batches)... [2026-10-22 10:06:42][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 10:07:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.9730 [2026-10-22 10:07:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2784 [2026-10-22 10:07:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.1524 [2026-10-22 10:07:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5029 [2026-10-22 10:07:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5553 [2026-10-22 10:07:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6551 [2026-10-22 10:07:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6273 [2026-10-22 10:07:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7664 [2026-10-22 10:07:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2964 [2026-10-22 10:07:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3285 [2026-10-22 10:07:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0625 [2026-10-22 10:07:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.7836 [2026-10-22 10:07:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=3.8720 [2026-10-22 10:07:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=3.8605 [2026-10-22 10:07:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=3.7671 [2026-10-22 10:07:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.0661 [2026-10-22 10:08:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9788 [2026-10-22 10:08:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8595 [2026-10-22 10:08:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.1730 [2026-10-22 10:08:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6178 [2026-10-22 10:08:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6443 [2026-10-22 10:08:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6315 [2026-10-22 10:08:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.1134 [2026-10-22 10:08:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3420 [2026-10-22 10:08:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4155 [2026-10-22 10:08:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8835 [2026-10-22 10:08:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7344 [2026-10-22 10:08:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9665 [2026-10-22 10:09:00][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,624 | MFU: 1.8% [2026-10-22 10:09:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2825 [2026-10-22 10:09:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9834 [2026-10-22 10:09:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1838 [2026-10-22 10:09:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7580 [2026-10-22 10:09:05][INFO][training.pretrain_engine] [Ph2|Step 686] loss=4.7542 | ppl=116.08 | aux=0.02508 | lr=2.97e-04 | gnorm=0.566 | tok/s=29,267 | MFU=7.0% | step=143311ms | tokens=1.521B | mem=4.2/32.7GB | cost=$56.29/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 18:04 (127.9h) | Phase3: SKIPPED | SFT: Oct 27 22:49 (132.7h) | DPO (Final): Oct 27 23:23 (133.2h) [2026-10-22 10:09:05][INFO][training.pretrain_engine] Starting global step 687 (accumulating 32 micro-batches)... [2026-10-22 10:09:05][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 10:09:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2304 [2026-10-22 10:09:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7665 [2026-10-22 10:09:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1677 [2026-10-22 10:09:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9849 [2026-10-22 10:09:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5943 [2026-10-22 10:09:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6405 [2026-10-22 10:09:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7390 [2026-10-22 10:09:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3945 [2026-10-22 10:09:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8481 [2026-10-22 10:09:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2766 [2026-10-22 10:09:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9635 [2026-10-22 10:09:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8279 [2026-10-22 10:10:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0487 [2026-10-22 10:10:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0928 [2026-10-22 10:10:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6197 [2026-10-22 10:10:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0482 [2026-10-22 10:10:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7249 [2026-10-22 10:10:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7411 [2026-10-22 10:10:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9142 [2026-10-22 10:10:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6330 [2026-10-22 10:10:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9098 [2026-10-22 10:10:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8476 [2026-10-22 10:10:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5606 [2026-10-22 10:10:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6348 [2026-10-22 10:11:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8730 [2026-10-22 10:11:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8159 [2026-10-22 10:11:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3167 [2026-10-22 10:11:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9925 [2026-10-22 10:11:23][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,609 | MFU: 1.8% [2026-10-22 10:11:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3235 [2026-10-22 10:11:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3068 [2026-10-22 10:11:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1580 [2026-10-22 10:11:28][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.8840 [2026-10-22 10:11:28][INFO][training.pretrain_engine] [Ph2|Step 687] loss=4.8369 | ppl=126.07 | aux=0.02505 | lr=2.97e-04 | gnorm=0.535 | tok/s=29,270 | MFU=7.0% | step=143297ms | tokens=1.523B | mem=4.2/32.7GB | cost=$56.37/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 18:05 (127.9h) | Phase3: SKIPPED | SFT: Oct 27 22:50 (132.6h) | DPO (Final): Oct 27 23:24 (133.2h) [2026-10-22 10:11:28][INFO][training.pretrain_engine] Starting global step 688 (accumulating 32 micro-batches)... [2026-10-22 10:11:28][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 10:11:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7711 [2026-10-22 10:11:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9204 [2026-10-22 10:11:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6799 [2026-10-22 10:11:46][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5768 [2026-10-22 10:12:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8889 [2026-10-22 10:12:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6125 [2026-10-22 10:12:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7647 [2026-10-22 10:12:04][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1318 [2026-10-22 10:12:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9329 [2026-10-22 10:12:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6783 [2026-10-22 10:12:21][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9248 [2026-10-22 10:12:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8884 [2026-10-22 10:12:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8004 [2026-10-22 10:12:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4850 [2026-10-22 10:12:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9499 [2026-10-22 10:12:39][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7304 [2026-10-22 10:12:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8898 [2026-10-22 10:12:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3409 [2026-10-22 10:12:57][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0152 [2026-10-22 10:12:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7956 [2026-10-22 10:13:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2576 [2026-10-22 10:13:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2860 [2026-10-22 10:13:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4063 [2026-10-22 10:13:15][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6635 [2026-10-22 10:13:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2791 [2026-10-22 10:13:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2379 [2026-10-22 10:13:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=3.7963 [2026-10-22 10:13:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2987 [2026-10-22 10:13:45][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,624 | MFU: 1.8% [2026-10-22 10:13:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8042 [2026-10-22 10:13:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9934 [2026-10-22 10:13:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0766 [2026-10-22 10:13:51][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4425 [2026-10-22 10:13:51][INFO][training.pretrain_engine] [Ph2|Step 688] loss=4.7321 | ppl=113.53 | aux=0.02506 | lr=2.97e-04 | gnorm=0.570 | tok/s=29,272 | MFU=7.0% | step=143287ms | tokens=1.524B | mem=4.2/32.7GB | cost=$56.46/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 18:06 (127.9h) | Phase3: SKIPPED | SFT: Oct 27 22:51 (132.6h) | DPO (Final): Oct 27 23:25 (133.2h) [2026-10-22 10:13:51][INFO][training.pretrain_engine] Starting global step 689 (accumulating 32 micro-batches)... [2026-10-22 10:13:51][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 10:14:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8691 [2026-10-22 10:14:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4337 [2026-10-22 10:14:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6147 [2026-10-22 10:14:09][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1700 [2026-10-22 10:14:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8837 [2026-10-22 10:14:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7920 [2026-10-22 10:14:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8832 [2026-10-22 10:14:27][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8028 [2026-10-22 10:14:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0757 [2026-10-22 10:14:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3364 [2026-10-22 10:14:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0075 [2026-10-22 10:14:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9364 [2026-10-22 10:15:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8825 [2026-10-22 10:15:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8887 [2026-10-22 10:15:02][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8404 [2026-10-22 10:15:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7236 [2026-10-22 10:15:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5546 [2026-10-22 10:15:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9962 [2026-10-22 10:15:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1011 [2026-10-22 10:15:20][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3624 [2026-10-22 10:15:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1764 [2026-10-22 10:15:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0508 [2026-10-22 10:15:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8967 [2026-10-22 10:15:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7286 [2026-10-22 10:15:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9489 [2026-10-22 10:15:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5555 [2026-10-22 10:15:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5834 [2026-10-22 10:15:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0888 [2026-10-22 10:16:09][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,628 | MFU: 1.8% [2026-10-22 10:16:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3632 [2026-10-22 10:16:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4764 [2026-10-22 10:16:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6225 [2026-10-22 10:16:14][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6215 [2026-10-22 10:16:14][INFO][training.pretrain_engine] [Ph2|Step 689] loss=4.9190 | ppl=136.87 | aux=0.02505 | lr=2.97e-04 | gnorm=0.621 | tok/s=29,274 | MFU=7.0% | step=143279ms | tokens=1.525B | mem=4.2/32.7GB | cost=$56.54/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 18:08 (127.9h) | Phase3: SKIPPED | SFT: Oct 27 22:52 (132.6h) | DPO (Final): Oct 27 23:27 (133.2h) [2026-10-22 10:16:14][INFO][training.pretrain_engine] Starting global step 690 (accumulating 32 micro-batches)... [2026-10-22 10:16:14][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 10:16:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1565 [2026-10-22 10:16:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4293 [2026-10-22 10:16:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0154 [2026-10-22 10:16:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5983 [2026-10-22 10:16:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5629 [2026-10-22 10:16:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7458 [2026-10-22 10:16:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0050 [2026-10-22 10:16:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2190 [2026-10-22 10:17:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3781 [2026-10-22 10:17:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8701 [2026-10-22 10:17:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6764 [2026-10-22 10:17:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9382 [2026-10-22 10:17:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8858 [2026-10-22 10:17:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3907 [2026-10-22 10:17:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.0666 [2026-10-22 10:17:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7266 [2026-10-22 10:17:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4250 [2026-10-22 10:17:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7702 [2026-10-22 10:17:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9031 [2026-10-22 10:17:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9105 [2026-10-22 10:18:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0488 [2026-10-22 10:18:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5596 [2026-10-22 10:18:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9216 [2026-10-22 10:18:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1162 [2026-10-22 10:18:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5697 [2026-10-22 10:18:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6175 [2026-10-22 10:18:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7421 [2026-10-22 10:18:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7260 [2026-10-22 10:18:31][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,621 | MFU: 1.8% [2026-10-22 10:18:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1708 [2026-10-22 10:18:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8271 [2026-10-22 10:18:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1994 [2026-10-22 10:18:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2317 [2026-10-22 10:18:37][INFO][training.pretrain_engine] [Ph2|Step 690] loss=4.7988 | ppl=121.37 | aux=0.02505 | lr=2.97e-04 | gnorm=0.566 | tok/s=29,276 | MFU=7.0% | step=143269ms | tokens=1.526B | mem=4.2/32.7GB | cost=$56.62/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 18:09 (127.8h) | Phase3: SKIPPED | SFT: Oct 27 22:54 (132.6h) | DPO (Final): Oct 27 23:28 (133.2h) [2026-10-22 10:18:37][INFO][training.pretrain_engine] Starting global step 691 (accumulating 32 micro-batches)... [2026-10-22 10:18:37][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 10:18:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8496 [2026-10-22 10:18:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9309 [2026-10-22 10:18:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9950 [2026-10-22 10:18:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0464 [2026-10-22 10:19:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5997 [2026-10-22 10:19:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.8429 [2026-10-22 10:19:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3012 [2026-10-22 10:19:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3744 [2026-10-22 10:19:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5273 [2026-10-22 10:19:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7857 [2026-10-22 10:19:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5778 [2026-10-22 10:19:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7107 [2026-10-22 10:19:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5266 [2026-10-22 10:19:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.0772 [2026-10-22 10:19:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.0152 [2026-10-22 10:19:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4559 [2026-10-22 10:20:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6594 [2026-10-22 10:20:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3714 [2026-10-22 10:20:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8898 [2026-10-22 10:20:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3808 [2026-10-22 10:20:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8922 [2026-10-22 10:20:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7792 [2026-10-22 10:20:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6427 [2026-10-22 10:20:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5418 [2026-10-22 10:20:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0555 [2026-10-22 10:20:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9847 [2026-10-22 10:20:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1083 [2026-10-22 10:20:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8324 [2026-10-22 10:20:54][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,611 | MFU: 1.8% [2026-10-22 10:20:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9510 [2026-10-22 10:20:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9411 [2026-10-22 10:20:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7234 [2026-10-22 10:20:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7011 [2026-10-22 10:20:59][INFO][training.pretrain_engine] [Ph2|Step 691] loss=4.6194 | ppl=101.44 | aux=0.02505 | lr=2.97e-04 | gnorm=0.660 | tok/s=29,278 | MFU=7.0% | step=143257ms | tokens=1.527B | mem=4.2/32.7GB | cost=$56.70/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 18:10 (127.8h) | Phase3: SKIPPED | SFT: Oct 27 22:55 (132.6h) | DPO (Final): Oct 27 23:29 (133.1h) [2026-10-22 10:20:59][INFO][training.pretrain_engine] Starting global step 692 (accumulating 32 micro-batches)... [2026-10-22 10:20:59][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 10:21:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1289 [2026-10-22 10:21:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8752 [2026-10-22 10:21:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1277 [2026-10-22 10:21:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0896 [2026-10-22 10:21:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7667 [2026-10-22 10:21:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1118 [2026-10-22 10:21:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1773 [2026-10-22 10:21:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1272 [2026-10-22 10:21:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8499 [2026-10-22 10:21:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8168 [2026-10-22 10:21:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9897 [2026-10-22 10:21:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7598 [2026-10-22 10:22:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4460 [2026-10-22 10:22:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8767 [2026-10-22 10:22:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5286 [2026-10-22 10:22:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7730 [2026-10-22 10:22:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2303 [2026-10-22 10:22:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0361 [2026-10-22 10:22:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6810 [2026-10-22 10:22:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0058 [2026-10-22 10:22:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3408 [2026-10-22 10:22:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.1049 [2026-10-22 10:22:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4167 [2026-10-22 10:22:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1541 [2026-10-22 10:23:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9008 [2026-10-22 10:23:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4452 [2026-10-22 10:23:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7290 [2026-10-22 10:23:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7267 [2026-10-22 10:23:17][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,607 | MFU: 1.8% [2026-10-22 10:23:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7055 [2026-10-22 10:23:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0478 [2026-10-22 10:23:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7950 [2026-10-22 10:23:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4583 [2026-10-22 10:23:22][INFO][training.pretrain_engine] [Ph2|Step 692] loss=4.8226 | ppl=124.29 | aux=0.02505 | lr=2.97e-04 | gnorm=0.590 | tok/s=29,280 | MFU=7.0% | step=143248ms | tokens=1.528B | mem=4.2/32.7GB | cost=$56.78/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 18:11 (127.8h) | Phase3: SKIPPED | SFT: Oct 27 22:56 (132.6h) | DPO (Final): Oct 27 23:30 (133.1h) [2026-10-22 10:23:22][INFO][training.pretrain_engine] Starting global step 693 (accumulating 32 micro-batches)... [2026-10-22 10:23:22][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 10:23:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6932 [2026-10-22 10:23:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8296 [2026-10-22 10:23:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7372 [2026-10-22 10:23:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7785 [2026-10-22 10:23:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5897 [2026-10-22 10:23:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7349 [2026-10-22 10:23:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7002 [2026-10-22 10:23:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0320 [2026-10-22 10:24:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5602 [2026-10-22 10:24:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8153 [2026-10-22 10:24:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.1088 [2026-10-22 10:24:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9226 [2026-10-22 10:24:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5965 [2026-10-22 10:24:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6319 [2026-10-22 10:24:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5747 [2026-10-22 10:24:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6090 [2026-10-22 10:24:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9014 [2026-10-22 10:24:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9617 [2026-10-22 10:24:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9298 [2026-10-22 10:24:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5549 [2026-10-22 10:25:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8741 [2026-10-22 10:25:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8790 [2026-10-22 10:25:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2230 [2026-10-22 10:25:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4999 [2026-10-22 10:25:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5074 [2026-10-22 10:25:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1820 [2026-10-22 10:25:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9393 [2026-10-22 10:25:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3350 [2026-10-22 10:25:39][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,625 | MFU: 1.8% [2026-10-22 10:25:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4718 [2026-10-22 10:25:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1999 [2026-10-22 10:25:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.6051 [2026-10-22 10:25:45][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3613 [2026-10-22 10:25:45][INFO][training.pretrain_engine] [Ph2|Step 693] loss=4.8061 | ppl=122.26 | aux=0.02506 | lr=2.97e-04 | gnorm=0.578 | tok/s=29,282 | MFU=7.0% | step=143240ms | tokens=1.529B | mem=4.2/32.7GB | cost=$56.86/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 18:13 (127.8h) | Phase3: SKIPPED | SFT: Oct 27 22:57 (132.5h) | DPO (Final): Oct 27 23:32 (133.1h) [2026-10-22 10:25:48][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step693 | Tokens: 1.529B | Spent: $56.87 | Remaining: $943.13 [2026-10-22 10:25:48][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step693 [2026-10-22 10:25:48][INFO][training.pretrain_engine] Starting global step 694 (accumulating 32 micro-batches)... [2026-10-22 10:25:48][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 10:25:48][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-22 10:25:48][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-22 10:25:48][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-22 10:25:48][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step693 to Phase-Technologies/forge-500m-pretrain (path: phase2_step693)... [2026-10-22 10:25:48][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-22 10:26:00][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-22 10:26:00][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step693 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-22 10:26:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3582 [2026-10-22 10:26:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3443 [2026-10-22 10:26:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4701 [2026-10-22 10:26:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.0406 [2026-10-22 10:26:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=6.0883 [2026-10-22 10:26:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3513 [2026-10-22 10:26:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1255 [2026-10-22 10:26:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6945 [2026-10-22 10:26:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5135 [2026-10-22 10:26:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0730 [2026-10-22 10:26:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3246 [2026-10-22 10:26:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2982 [2026-10-22 10:26:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1776 [2026-10-22 10:26:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8825 [2026-10-22 10:26:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0985 [2026-10-22 10:26:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7236 [2026-10-22 10:27:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5816 [2026-10-22 10:27:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9173 [2026-10-22 10:27:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9317 [2026-10-22 10:27:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7022 [2026-10-22 10:27:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6990 [2026-10-22 10:27:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4262 [2026-10-22 10:27:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4188 [2026-10-22 10:27:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2226 [2026-10-22 10:27:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.1857 [2026-10-22 10:27:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=3.9962 [2026-10-22 10:27:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.0878 [2026-10-22 10:27:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2334 [2026-10-22 10:28:05][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,613 | MFU: 1.8% [2026-10-22 10:28:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6714 [2026-10-22 10:28:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6491 [2026-10-22 10:28:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7303 [2026-10-22 10:28:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8640 [2026-10-22 10:28:10][INFO][training.pretrain_engine] [Ph2|Step 694] loss=4.7242 | ppl=112.64 | aux=0.02506 | lr=2.97e-04 | gnorm=0.551 | tok/s=29,284 | MFU=7.0% | step=143228ms | tokens=1.530B | mem=4.2/32.7GB | cost=$56.95/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 18:14 (127.8h) | Phase3: SKIPPED | SFT: Oct 27 22:59 (132.5h) | DPO (Final): Oct 27 23:33 (133.1h) [2026-10-22 10:28:10][INFO][training.pretrain_engine] Starting global step 695 (accumulating 32 micro-batches)... [2026-10-22 10:28:10][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 10:28:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6524 [2026-10-22 10:28:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4131 [2026-10-22 10:28:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1082 [2026-10-22 10:28:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1453 [2026-10-22 10:28:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6825 [2026-10-22 10:28:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7417 [2026-10-22 10:28:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4201 [2026-10-22 10:28:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.0112 [2026-10-22 10:29:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3683 [2026-10-22 10:29:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9754 [2026-10-22 10:29:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0338 [2026-10-22 10:29:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5255 [2026-10-22 10:29:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6358 [2026-10-22 10:29:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0028 [2026-10-22 10:29:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7485 [2026-10-22 10:29:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6181 [2026-10-22 10:29:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4332 [2026-10-22 10:29:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7948 [2026-10-22 10:29:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.7875 [2026-10-22 10:29:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0838 [2026-10-22 10:29:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=3.8195 [2026-10-22 10:29:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6951 [2026-10-22 10:29:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7434 [2026-10-22 10:29:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7314 [2026-10-22 10:30:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8565 [2026-10-22 10:30:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8972 [2026-10-22 10:30:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7829 [2026-10-22 10:30:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9809 [2026-10-22 10:30:27][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,621 | MFU: 1.8% [2026-10-22 10:30:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7188 [2026-10-22 10:30:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9176 [2026-10-22 10:30:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4711 [2026-10-22 10:30:32][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5870 [2026-10-22 10:30:32][INFO][training.pretrain_engine] [Ph2|Step 695] loss=4.8568 | ppl=128.61 | aux=0.02505 | lr=2.97e-04 | gnorm=0.471 | tok/s=29,285 | MFU=7.0% | step=143225ms | tokens=1.531B | mem=4.2/32.7GB | cost=$57.03/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 18:16 (127.8h) | Phase3: SKIPPED | SFT: Oct 27 23:00 (132.5h) | DPO (Final): Oct 27 23:34 (133.1h) [2026-10-22 10:30:32][INFO][training.pretrain_engine] Starting global step 696 (accumulating 32 micro-batches)... [2026-10-22 10:30:32][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 10:30:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7765 [2026-10-22 10:30:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1085 [2026-10-22 10:30:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8448 [2026-10-22 10:30:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8331 [2026-10-22 10:31:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0296 [2026-10-22 10:31:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2511 [2026-10-22 10:31:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2951 [2026-10-22 10:31:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2909 [2026-10-22 10:31:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8139 [2026-10-22 10:31:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9543 [2026-10-22 10:31:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0314 [2026-10-22 10:31:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8121 [2026-10-22 10:31:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8403 [2026-10-22 10:31:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7326 [2026-10-22 10:31:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0188 [2026-10-22 10:31:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4435 [2026-10-22 10:32:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1075 [2026-10-22 10:32:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0917 [2026-10-22 10:32:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9397 [2026-10-22 10:32:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4358 [2026-10-22 10:32:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3939 [2026-10-22 10:32:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6728 [2026-10-22 10:32:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5163 [2026-10-22 10:32:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4997 [2026-10-22 10:32:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6461 [2026-10-22 10:32:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6630 [2026-10-22 10:32:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4769 [2026-10-22 10:32:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3609 [2026-10-22 10:32:50][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,602 | MFU: 1.8% [2026-10-22 10:32:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5900 [2026-10-22 10:32:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9130 [2026-10-22 10:32:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5993 [2026-10-22 10:32:55][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4824 [2026-10-22 10:32:55][INFO][training.pretrain_engine] [Ph2|Step 696] loss=4.7356 | ppl=113.93 | aux=0.02506 | lr=2.97e-04 | gnorm=0.539 | tok/s=29,287 | MFU=7.0% | step=143212ms | tokens=1.532B | mem=4.2/32.7GB | cost=$57.11/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 18:17 (127.7h) | Phase3: SKIPPED | SFT: Oct 27 23:01 (132.5h) | DPO (Final): Oct 27 23:35 (133.0h) [2026-10-22 10:32:55][INFO][training.pretrain_engine] Starting global step 697 (accumulating 32 micro-batches)... [2026-10-22 10:32:55][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 10:33:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9980 [2026-10-22 10:33:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8885 [2026-10-22 10:33:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8113 [2026-10-22 10:33:13][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2790 [2026-10-22 10:33:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5666 [2026-10-22 10:33:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5951 [2026-10-22 10:33:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6081 [2026-10-22 10:33:31][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6968 [2026-10-22 10:33:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1135 [2026-10-22 10:33:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2089 [2026-10-22 10:33:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0828 [2026-10-22 10:33:49][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1012 [2026-10-22 10:34:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4523 [2026-10-22 10:34:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3303 [2026-10-22 10:34:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=3.6756 [2026-10-22 10:34:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2657 [2026-10-22 10:34:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5827 [2026-10-22 10:34:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1134 [2026-10-22 10:34:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5745 [2026-10-22 10:34:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6946 [2026-10-22 10:34:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5158 [2026-10-22 10:34:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2384 [2026-10-22 10:34:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0191 [2026-10-22 10:34:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8738 [2026-10-22 10:34:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6240 [2026-10-22 10:34:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8096 [2026-10-22 10:35:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6121 [2026-10-22 10:35:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7247 [2026-10-22 10:35:13][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,612 | MFU: 1.8% [2026-10-22 10:35:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9746 [2026-10-22 10:35:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.9476 [2026-10-22 10:35:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9596 [2026-10-22 10:35:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3771 [2026-10-22 10:35:18][INFO][training.pretrain_engine] [Ph2|Step 697] loss=4.7122 | ppl=111.29 | aux=0.02505 | lr=2.97e-04 | gnorm=0.559 | tok/s=29,288 | MFU=7.0% | step=143209ms | tokens=1.533B | mem=4.2/32.7GB | cost=$57.19/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 18:18 (127.7h) | Phase3: SKIPPED | SFT: Oct 27 23:03 (132.5h) | DPO (Final): Oct 27 23:37 (133.0h) [2026-10-22 10:35:18][INFO][training.pretrain_engine] Starting global step 698 (accumulating 32 micro-batches)... [2026-10-22 10:35:18][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 10:35:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5656 [2026-10-22 10:35:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.0450 [2026-10-22 10:35:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2581 [2026-10-22 10:35:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6927 [2026-10-22 10:35:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1538 [2026-10-22 10:35:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4125 [2026-10-22 10:35:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6280 [2026-10-22 10:35:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4176 [2026-10-22 10:36:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6243 [2026-10-22 10:36:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7967 [2026-10-22 10:36:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3483 [2026-10-22 10:36:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8757 [2026-10-22 10:36:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5586 [2026-10-22 10:36:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8190 [2026-10-22 10:36:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7181 [2026-10-22 10:36:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5119 [2026-10-22 10:36:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7208 [2026-10-22 10:36:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7008 [2026-10-22 10:36:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6598 [2026-10-22 10:36:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9181 [2026-10-22 10:37:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7721 [2026-10-22 10:37:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9619 [2026-10-22 10:37:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0712 [2026-10-22 10:37:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7343 [2026-10-22 10:37:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9839 [2026-10-22 10:37:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0976 [2026-10-22 10:37:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1431 [2026-10-22 10:37:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5374 [2026-10-22 10:37:36][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,610 | MFU: 1.8% [2026-10-22 10:37:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3551 [2026-10-22 10:37:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5903 [2026-10-22 10:37:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4337 [2026-10-22 10:37:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.0874 [2026-10-22 10:37:41][INFO][training.pretrain_engine] [Ph2|Step 698] loss=4.7494 | ppl=115.51 | aux=0.02505 | lr=2.97e-04 | gnorm=0.562 | tok/s=29,290 | MFU=7.0% | step=143200ms | tokens=1.534B | mem=4.2/32.7GB | cost=$57.27/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 18:20 (127.7h) | Phase3: SKIPPED | SFT: Oct 27 23:04 (132.4h) | DPO (Final): Oct 27 23:38 (133.0h) [2026-10-22 10:37:41][INFO][training.pretrain_engine] Starting global step 699 (accumulating 32 micro-batches)... [2026-10-22 10:37:41][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 10:37:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0281 [2026-10-22 10:37:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6864 [2026-10-22 10:37:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1495 [2026-10-22 10:37:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8631 [2026-10-22 10:38:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8796 [2026-10-22 10:38:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3787 [2026-10-22 10:38:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4752 [2026-10-22 10:38:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6022 [2026-10-22 10:38:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3203 [2026-10-22 10:38:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7719 [2026-10-22 10:38:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3754 [2026-10-22 10:38:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5121 [2026-10-22 10:38:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3061 [2026-10-22 10:38:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0304 [2026-10-22 10:38:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2354 [2026-10-22 10:38:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1468 [2026-10-22 10:39:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=3.8458 [2026-10-22 10:39:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.1177 [2026-10-22 10:39:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9441 [2026-10-22 10:39:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4309 [2026-10-22 10:39:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3093 [2026-10-22 10:39:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6874 [2026-10-22 10:39:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5739 [2026-10-22 10:39:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9356 [2026-10-22 10:39:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.6322 [2026-10-22 10:39:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6374 [2026-10-22 10:39:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1859 [2026-10-22 10:39:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8898 [2026-10-22 10:39:59][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,618 | MFU: 1.8% [2026-10-22 10:40:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7236 [2026-10-22 10:40:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8484 [2026-10-22 10:40:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6515 [2026-10-22 10:40:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8385 [2026-10-22 10:40:04][INFO][training.pretrain_engine] [Ph2|Step 699] loss=4.8950 | ppl=133.62 | aux=0.02505 | lr=2.97e-04 | gnorm=0.594 | tok/s=29,291 | MFU=7.0% | step=143192ms | tokens=1.535B | mem=4.2/32.7GB | cost=$57.36/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 18:21 (127.7h) | Phase3: SKIPPED | SFT: Oct 27 23:06 (132.4h) | DPO (Final): Oct 27 23:40 (133.0h) [2026-10-22 10:40:04][INFO][training.pretrain_engine] Starting global step 700 (accumulating 32 micro-batches)... [2026-10-22 10:40:04][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 10:40:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0293 [2026-10-22 10:40:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7227 [2026-10-22 10:40:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8460 [2026-10-22 10:40:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8249 [2026-10-22 10:40:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5971 [2026-10-22 10:40:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3770 [2026-10-22 10:40:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4739 [2026-10-22 10:40:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2696 [2026-10-22 10:40:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8053 [2026-10-22 10:40:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5483 [2026-10-22 10:40:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0916 [2026-10-22 10:40:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2956 [2026-10-22 10:41:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2037 [2026-10-22 10:41:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.0443 [2026-10-22 10:41:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6593 [2026-10-22 10:41:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3984 [2026-10-22 10:41:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7735 [2026-10-22 10:41:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1362 [2026-10-22 10:41:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4156 [2026-10-22 10:41:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1146 [2026-10-22 10:41:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0275 [2026-10-22 10:41:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4748 [2026-10-22 10:41:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7944 [2026-10-22 10:41:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8067 [2026-10-22 10:42:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8094 [2026-10-22 10:42:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6955 [2026-10-22 10:42:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8866 [2026-10-22 10:42:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8320 [2026-10-22 10:42:21][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,619 | MFU: 1.8% [2026-10-22 10:42:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9019 [2026-10-22 10:42:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8206 [2026-10-22 10:42:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4683 [2026-10-22 10:42:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7226 [2026-10-22 10:42:26][INFO][training.pretrain_engine] [Ph2|Step 700] loss=4.8373 | ppl=126.13 | aux=0.02505 | lr=2.97e-04 | gnorm=0.590 | tok/s=29,293 | MFU=7.0% | step=143184ms | tokens=1.536B | mem=4.2/32.7GB | cost=$57.44/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 18:22 (127.7h) | Phase3: SKIPPED | SFT: Oct 27 23:07 (132.4h) | DPO (Final): Oct 27 23:41 (133.0h) [2026-10-22 10:42:26][INFO][training.pretrain_engine] Starting global step 701 (accumulating 32 micro-batches)... [2026-10-22 10:42:26][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 10:42:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5366 [2026-10-22 10:42:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0589 [2026-10-22 10:42:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9975 [2026-10-22 10:42:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0728 [2026-10-22 10:43:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6112 [2026-10-22 10:43:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6717 [2026-10-22 10:43:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6493 [2026-10-22 10:43:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9598 [2026-10-22 10:43:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1767 [2026-10-22 10:43:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6517 [2026-10-22 10:43:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0308 [2026-10-22 10:43:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6829 [2026-10-22 10:43:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0766 [2026-10-22 10:43:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1026 [2026-10-22 10:43:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9936 [2026-10-22 10:43:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0469 [2026-10-22 10:43:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1346 [2026-10-22 10:43:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2617 [2026-10-22 10:43:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2322 [2026-10-22 10:43:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8715 [2026-10-22 10:44:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5413 [2026-10-22 10:44:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7250 [2026-10-22 10:44:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4233 [2026-10-22 10:44:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6761 [2026-10-22 10:44:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6954 [2026-10-22 10:44:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9873 [2026-10-22 10:44:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6068 [2026-10-22 10:44:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7294 [2026-10-22 10:44:44][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,604 | MFU: 1.8% [2026-10-22 10:44:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7022 [2026-10-22 10:44:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6340 [2026-10-22 10:44:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2749 [2026-10-22 10:44:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8435 [2026-10-22 10:44:49][INFO][training.pretrain_engine] [Ph2|Step 701] loss=4.8474 | ppl=127.41 | aux=0.02506 | lr=2.97e-04 | gnorm=0.629 | tok/s=29,294 | MFU=7.0% | step=143178ms | tokens=1.537B | mem=4.2/32.7GB | cost=$57.52/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 18:24 (127.7h) | Phase3: SKIPPED | SFT: Oct 27 23:08 (132.4h) | DPO (Final): Oct 27 23:42 (133.0h) [2026-10-22 10:44:49][INFO][training.pretrain_engine] Starting global step 702 (accumulating 32 micro-batches)... [2026-10-22 10:44:49][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 10:45:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2393 [2026-10-22 10:45:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6451 [2026-10-22 10:45:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9706 [2026-10-22 10:45:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8184 [2026-10-22 10:45:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9447 [2026-10-22 10:45:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7807 [2026-10-22 10:45:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9617 [2026-10-22 10:45:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6521 [2026-10-22 10:45:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1343 [2026-10-22 10:45:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7130 [2026-10-22 10:45:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0176 [2026-10-22 10:45:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0810 [2026-10-22 10:46:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5093 [2026-10-22 10:46:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5924 [2026-10-22 10:46:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4236 [2026-10-22 10:46:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2353 [2026-10-22 10:46:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3542 [2026-10-22 10:46:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8189 [2026-10-22 10:46:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3603 [2026-10-22 10:46:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.2566 [2026-10-22 10:46:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6265 [2026-10-22 10:46:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8230 [2026-10-22 10:46:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4474 [2026-10-22 10:46:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5056 [2026-10-22 10:46:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=3.9943 [2026-10-22 10:46:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9214 [2026-10-22 10:46:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8478 [2026-10-22 10:46:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0972 [2026-10-22 10:47:07][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,585 | MFU: 1.8% [2026-10-22 10:47:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0981 [2026-10-22 10:47:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4593 [2026-10-22 10:47:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7334 [2026-10-22 10:47:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1906 [2026-10-22 10:47:12][INFO][training.pretrain_engine] [Ph2|Step 702] loss=4.7834 | ppl=119.51 | aux=0.02504 | lr=2.97e-04 | gnorm=0.789 | tok/s=29,295 | MFU=7.0% | step=143173ms | tokens=1.538B | mem=4.2/32.7GB | cost=$57.60/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 18:25 (127.6h) | Phase3: SKIPPED | SFT: Oct 27 23:10 (132.4h) | DPO (Final): Oct 27 23:44 (133.0h) [2026-10-22 10:47:15][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step702 | Tokens: 1.538B | Spent: $57.60 | Remaining: $942.40 [2026-10-22 10:47:15][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step702 [2026-10-22 10:47:15][INFO][training.pretrain_engine] Starting global step 703 (accumulating 32 micro-batches)... [2026-10-22 10:47:15][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 10:47:15][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-22 10:47:15][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-22 10:47:15][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-22 10:47:15][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step702 to Phase-Technologies/forge-500m-pretrain (path: phase2_step702)... [2026-10-22 10:47:15][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-22 10:47:24][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-22 10:47:24][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step702 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-22 10:47:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1317 [2026-10-22 10:47:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9841 [2026-10-22 10:47:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2093 [2026-10-22 10:47:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2889 [2026-10-22 10:47:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6677 [2026-10-22 10:47:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0642 [2026-10-22 10:47:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9265 [2026-10-22 10:47:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9132 [2026-10-22 10:48:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2568 [2026-10-22 10:48:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5310 [2026-10-22 10:48:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.1491 [2026-10-22 10:48:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7516 [2026-10-22 10:48:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8681 [2026-10-22 10:48:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9931 [2026-10-22 10:48:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8105 [2026-10-22 10:48:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6321 [2026-10-22 10:48:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9913 [2026-10-22 10:48:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4387 [2026-10-22 10:48:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6565 [2026-10-22 10:48:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9800 [2026-10-22 10:48:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.1163 [2026-10-22 10:48:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=3.8647 [2026-10-22 10:48:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9305 [2026-10-22 10:49:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=3.8773 [2026-10-22 10:49:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2039 [2026-10-22 10:49:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4430 [2026-10-22 10:49:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9499 [2026-10-22 10:49:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0564 [2026-10-22 10:49:33][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,589 | MFU: 1.8% [2026-10-22 10:49:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5713 [2026-10-22 10:49:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6941 [2026-10-22 10:49:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2346 [2026-10-22 10:49:37][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2322 [2026-10-22 10:49:37][INFO][training.pretrain_engine] [Ph2|Step 703] loss=4.6993 | ppl=109.87 | aux=0.02506 | lr=2.97e-04 | gnorm=0.887 | tok/s=29,298 | MFU=7.0% | step=143161ms | tokens=1.539B | mem=4.2/32.7GB | cost=$57.68/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 18:27 (127.6h) | Phase3: SKIPPED | SFT: Oct 27 23:11 (132.4h) | DPO (Final): Oct 27 23:45 (132.9h) [2026-10-22 10:49:37][INFO][training.pretrain_engine] Starting global step 704 (accumulating 32 micro-batches)... [2026-10-22 10:49:37][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 10:49:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5095 [2026-10-22 10:49:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3128 [2026-10-22 10:49:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1953 [2026-10-22 10:49:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8105 [2026-10-22 10:50:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6479 [2026-10-22 10:50:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0334 [2026-10-22 10:50:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1039 [2026-10-22 10:50:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4380 [2026-10-22 10:50:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5990 [2026-10-22 10:50:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=3.9078 [2026-10-22 10:50:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0026 [2026-10-22 10:50:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7033 [2026-10-22 10:50:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6211 [2026-10-22 10:50:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6584 [2026-10-22 10:50:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8585 [2026-10-22 10:50:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8776 [2026-10-22 10:51:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7878 [2026-10-22 10:51:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5328 [2026-10-22 10:51:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6987 [2026-10-22 10:51:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5001 [2026-10-22 10:51:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4029 [2026-10-22 10:51:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5478 [2026-10-22 10:51:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5352 [2026-10-22 10:51:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.1840 [2026-10-22 10:51:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4711 [2026-10-22 10:51:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7867 [2026-10-22 10:51:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6081 [2026-10-22 10:51:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6981 [2026-10-22 10:51:55][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,612 | MFU: 1.8% [2026-10-22 10:52:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7619 [2026-10-22 10:52:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4497 [2026-10-22 10:52:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6387 [2026-10-22 10:52:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6116 [2026-10-22 10:52:00][INFO][training.pretrain_engine] [Ph2|Step 704] loss=4.8226 | ppl=124.29 | aux=0.02505 | lr=2.97e-04 | gnorm=0.805 | tok/s=29,298 | MFU=7.0% | step=143161ms | tokens=1.540B | mem=4.2/32.7GB | cost=$57.77/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 18:28 (127.6h) | Phase3: SKIPPED | SFT: Oct 27 23:13 (132.4h) | DPO (Final): Oct 27 23:47 (132.9h) [2026-10-22 10:52:00][INFO][training.pretrain_engine] Starting global step 705 (accumulating 32 micro-batches)... [2026-10-22 10:52:00][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 10:52:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9539 [2026-10-22 10:52:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0299 [2026-10-22 10:52:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9176 [2026-10-22 10:52:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6165 [2026-10-22 10:52:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1393 [2026-10-22 10:52:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1268 [2026-10-22 10:52:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9738 [2026-10-22 10:52:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9488 [2026-10-22 10:52:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4625 [2026-10-22 10:52:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5391 [2026-10-22 10:52:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.1180 [2026-10-22 10:52:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2210 [2026-10-22 10:53:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8245 [2026-10-22 10:53:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8835 [2026-10-22 10:53:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5798 [2026-10-22 10:53:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7761 [2026-10-22 10:53:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1402 [2026-10-22 10:53:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1565 [2026-10-22 10:53:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7518 [2026-10-22 10:53:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9097 [2026-10-22 10:53:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0196 [2026-10-22 10:53:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6296 [2026-10-22 10:53:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.7225 [2026-10-22 10:53:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7524 [2026-10-22 10:54:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0660 [2026-10-22 10:54:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2041 [2026-10-22 10:54:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8378 [2026-10-22 10:54:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9787 [2026-10-22 10:54:18][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,627 | MFU: 1.8% [2026-10-22 10:54:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8924 [2026-10-22 10:54:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8634 [2026-10-22 10:54:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9370 [2026-10-22 10:54:24][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9750 [2026-10-22 10:54:24][INFO][training.pretrain_engine] [Ph2|Step 705] loss=4.7955 | ppl=120.96 | aux=0.02506 | lr=2.97e-04 | gnorm=0.742 | tok/s=29,299 | MFU=7.0% | step=143153ms | tokens=1.541B | mem=4.2/32.7GB | cost=$57.85/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 18:30 (127.6h) | Phase3: SKIPPED | SFT: Oct 27 23:14 (132.3h) | DPO (Final): Oct 27 23:48 (132.9h) [2026-10-22 10:54:24][INFO][training.pretrain_engine] Starting global step 706 (accumulating 32 micro-batches)... [2026-10-22 10:54:24][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 10:54:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.8844 [2026-10-22 10:54:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6171 [2026-10-22 10:54:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7086 [2026-10-22 10:54:42][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2146 [2026-10-22 10:54:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0028 [2026-10-22 10:54:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0705 [2026-10-22 10:54:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1670 [2026-10-22 10:55:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7817 [2026-10-22 10:55:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3961 [2026-10-22 10:55:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8276 [2026-10-22 10:55:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3067 [2026-10-22 10:55:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6211 [2026-10-22 10:55:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0130 [2026-10-22 10:55:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4385 [2026-10-22 10:55:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5863 [2026-10-22 10:55:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3965 [2026-10-22 10:55:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.2237 [2026-10-22 10:55:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4022 [2026-10-22 10:55:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3345 [2026-10-22 10:55:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3786 [2026-10-22 10:56:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9409 [2026-10-22 10:56:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7283 [2026-10-22 10:56:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8149 [2026-10-22 10:56:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3844 [2026-10-22 10:56:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5803 [2026-10-22 10:56:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.1689 [2026-10-22 10:56:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4829 [2026-10-22 10:56:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2599 [2026-10-22 10:56:41][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,614 | MFU: 1.8% [2026-10-22 10:56:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8724 [2026-10-22 10:56:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8759 [2026-10-22 10:56:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0267 [2026-10-22 10:56:47][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6410 [2026-10-22 10:56:47][INFO][training.pretrain_engine] [Ph2|Step 706] loss=4.8696 | ppl=130.27 | aux=0.02505 | lr=2.97e-04 | gnorm=0.664 | tok/s=29,300 | MFU=7.0% | step=143149ms | tokens=1.542B | mem=4.2/32.7GB | cost=$57.93/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 18:31 (127.6h) | Phase3: SKIPPED | SFT: Oct 27 23:16 (132.3h) | DPO (Final): Oct 27 23:50 (132.9h) [2026-10-22 10:56:47][INFO][training.pretrain_engine] Starting global step 707 (accumulating 32 micro-batches)... [2026-10-22 10:56:47][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 10:57:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0076 [2026-10-22 10:57:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9162 [2026-10-22 10:57:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7998 [2026-10-22 10:57:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6970 [2026-10-22 10:57:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6730 [2026-10-22 10:57:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3655 [2026-10-22 10:57:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6217 [2026-10-22 10:57:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.0634 [2026-10-22 10:57:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3751 [2026-10-22 10:57:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7585 [2026-10-22 10:57:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4964 [2026-10-22 10:57:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7371 [2026-10-22 10:57:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4987 [2026-10-22 10:57:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1293 [2026-10-22 10:57:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9473 [2026-10-22 10:57:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0984 [2026-10-22 10:58:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0497 [2026-10-22 10:58:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6515 [2026-10-22 10:58:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9814 [2026-10-22 10:58:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4214 [2026-10-22 10:58:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5521 [2026-10-22 10:58:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3833 [2026-10-22 10:58:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4531 [2026-10-22 10:58:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.1902 [2026-10-22 10:58:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5173 [2026-10-22 10:58:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4278 [2026-10-22 10:58:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.0540 [2026-10-22 10:58:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2614 [2026-10-22 10:59:05][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,610 | MFU: 1.8% [2026-10-22 10:59:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8956 [2026-10-22 10:59:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7606 [2026-10-22 10:59:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7713 [2026-10-22 10:59:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6281 [2026-10-22 10:59:10][INFO][training.pretrain_engine] [Ph2|Step 707] loss=4.7783 | ppl=118.90 | aux=0.02506 | lr=2.97e-04 | gnorm=0.629 | tok/s=29,303 | MFU=7.0% | step=143136ms | tokens=1.544B | mem=4.2/32.7GB | cost=$58.01/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 18:32 (127.6h) | Phase3: SKIPPED | SFT: Oct 27 23:17 (132.3h) | DPO (Final): Oct 27 23:51 (132.9h) [2026-10-22 10:59:10][INFO][training.pretrain_engine] Starting global step 708 (accumulating 32 micro-batches)... [2026-10-22 10:59:10][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 10:59:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2139 [2026-10-22 10:59:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8612 [2026-10-22 10:59:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2653 [2026-10-22 10:59:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9880 [2026-10-22 10:59:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2486 [2026-10-22 10:59:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9275 [2026-10-22 10:59:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2816 [2026-10-22 10:59:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1250 [2026-10-22 11:00:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5707 [2026-10-22 11:00:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9499 [2026-10-22 11:00:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5197 [2026-10-22 11:00:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.1954 [2026-10-22 11:00:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1903 [2026-10-22 11:00:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.8459 [2026-10-22 11:00:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0582 [2026-10-22 11:00:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4156 [2026-10-22 11:00:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.0589 [2026-10-22 11:00:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5911 [2026-10-22 11:00:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6506 [2026-10-22 11:00:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7313 [2026-10-22 11:00:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8683 [2026-10-22 11:00:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2554 [2026-10-22 11:00:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9703 [2026-10-22 11:00:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3485 [2026-10-22 11:01:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1179 [2026-10-22 11:01:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3316 [2026-10-22 11:01:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1363 [2026-10-22 11:01:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0140 [2026-10-22 11:01:28][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,611 | MFU: 1.8% [2026-10-22 11:01:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1550 [2026-10-22 11:01:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8624 [2026-10-22 11:01:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8990 [2026-10-22 11:01:33][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0393 [2026-10-22 11:01:33][INFO][training.pretrain_engine] [Ph2|Step 708] loss=4.7967 | ppl=121.10 | aux=0.02505 | lr=2.97e-04 | gnorm=0.547 | tok/s=29,304 | MFU=7.0% | step=143130ms | tokens=1.545B | mem=4.2/32.7GB | cost=$58.09/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 18:34 (127.5h) | Phase3: SKIPPED | SFT: Oct 27 23:18 (132.3h) | DPO (Final): Oct 27 23:52 (132.9h) [2026-10-22 11:01:33][INFO][training.pretrain_engine] Starting global step 709 (accumulating 32 micro-batches)... [2026-10-22 11:01:33][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 11:01:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5911 [2026-10-22 11:01:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6548 [2026-10-22 11:01:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3826 [2026-10-22 11:01:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2633 [2026-10-22 11:02:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7175 [2026-10-22 11:02:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8313 [2026-10-22 11:02:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7788 [2026-10-22 11:02:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9389 [2026-10-22 11:02:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0796 [2026-10-22 11:02:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8360 [2026-10-22 11:02:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4804 [2026-10-22 11:02:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1543 [2026-10-22 11:02:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8561 [2026-10-22 11:02:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2883 [2026-10-22 11:02:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0623 [2026-10-22 11:02:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2586 [2026-10-22 11:03:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7831 [2026-10-22 11:03:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9621 [2026-10-22 11:03:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7063 [2026-10-22 11:03:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4680 [2026-10-22 11:03:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4288 [2026-10-22 11:03:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7771 [2026-10-22 11:03:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3861 [2026-10-22 11:03:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6255 [2026-10-22 11:03:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8246 [2026-10-22 11:03:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9441 [2026-10-22 11:03:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9049 [2026-10-22 11:03:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7001 [2026-10-22 11:03:51][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,596 | MFU: 1.8% [2026-10-22 11:03:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7819 [2026-10-22 11:03:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9356 [2026-10-22 11:03:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8245 [2026-10-22 11:03:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7827 [2026-10-22 11:03:57][INFO][training.pretrain_engine] [Ph2|Step 709] loss=4.7476 | ppl=115.31 | aux=0.02505 | lr=2.97e-04 | gnorm=0.605 | tok/s=29,305 | MFU=7.0% | step=143125ms | tokens=1.546B | mem=4.2/32.7GB | cost=$58.18/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 18:35 (127.5h) | Phase3: SKIPPED | SFT: Oct 27 23:20 (132.3h) | DPO (Final): Oct 27 23:54 (132.8h) [2026-10-22 11:03:57][INFO][training.pretrain_engine] Starting global step 710 (accumulating 32 micro-batches)... [2026-10-22 11:03:57][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 11:04:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9848 [2026-10-22 11:04:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1776 [2026-10-22 11:04:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0213 [2026-10-22 11:04:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9271 [2026-10-22 11:04:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5182 [2026-10-22 11:04:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7284 [2026-10-22 11:04:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.0607 [2026-10-22 11:04:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2793 [2026-10-22 11:04:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5762 [2026-10-22 11:04:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.1994 [2026-10-22 11:04:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4146 [2026-10-22 11:04:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7020 [2026-10-22 11:05:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7784 [2026-10-22 11:05:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9685 [2026-10-22 11:05:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2394 [2026-10-22 11:05:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5856 [2026-10-22 11:05:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8622 [2026-10-22 11:05:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8882 [2026-10-22 11:05:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8502 [2026-10-22 11:05:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9030 [2026-10-22 11:05:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0655 [2026-10-22 11:05:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0085 [2026-10-22 11:05:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3439 [2026-10-22 11:05:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9277 [2026-10-22 11:06:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3966 [2026-10-22 11:06:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7877 [2026-10-22 11:06:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3365 [2026-10-22 11:06:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7635 [2026-10-22 11:06:14][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,606 | MFU: 1.8% [2026-10-22 11:06:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2462 [2026-10-22 11:06:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=3.8467 [2026-10-22 11:06:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.1069 [2026-10-22 11:06:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2770 [2026-10-22 11:06:19][INFO][training.pretrain_engine] [Ph2|Step 710] loss=4.7183 | ppl=111.98 | aux=0.02505 | lr=2.97e-04 | gnorm=0.617 | tok/s=29,307 | MFU=7.0% | step=143117ms | tokens=1.547B | mem=4.2/32.7GB | cost=$58.26/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 18:37 (127.5h) | Phase3: SKIPPED | SFT: Oct 27 23:21 (132.3h) | DPO (Final): Oct 27 23:55 (132.8h) [2026-10-22 11:06:20][INFO][training.pretrain_engine] Starting global step 711 (accumulating 32 micro-batches)... [2026-10-22 11:06:20][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 11:06:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7041 [2026-10-22 11:06:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2430 [2026-10-22 11:06:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5205 [2026-10-22 11:06:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9125 [2026-10-22 11:06:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9333 [2026-10-22 11:06:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8575 [2026-10-22 11:06:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8130 [2026-10-22 11:06:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6984 [2026-10-22 11:07:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4220 [2026-10-22 11:07:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5425 [2026-10-22 11:07:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6529 [2026-10-22 11:07:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4548 [2026-10-22 11:07:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0296 [2026-10-22 11:07:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9998 [2026-10-22 11:07:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8227 [2026-10-22 11:07:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2638 [2026-10-22 11:07:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1551 [2026-10-22 11:07:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5830 [2026-10-22 11:07:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1265 [2026-10-22 11:07:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7538 [2026-10-22 11:08:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9331 [2026-10-22 11:08:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.1189 [2026-10-22 11:08:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6951 [2026-10-22 11:08:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6187 [2026-10-22 11:08:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4853 [2026-10-22 11:08:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0011 [2026-10-22 11:08:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5536 [2026-10-22 11:08:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1013 [2026-10-22 11:08:37][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,610 | MFU: 1.8% [2026-10-22 11:08:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8600 [2026-10-22 11:08:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7699 [2026-10-22 11:08:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7278 [2026-10-22 11:08:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9933 [2026-10-22 11:08:42][INFO][training.pretrain_engine] [Ph2|Step 711] loss=4.7818 | ppl=119.32 | aux=0.02504 | lr=2.97e-04 | gnorm=0.664 | tok/s=29,309 | MFU=7.0% | step=143106ms | tokens=1.548B | mem=4.2/32.7GB | cost=$58.34/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 18:38 (127.5h) | Phase3: SKIPPED | SFT: Oct 27 23:22 (132.2h) | DPO (Final): Oct 27 23:56 (132.8h) [2026-10-22 11:08:46][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step711 | Tokens: 1.548B | Spent: $58.34 | Remaining: $941.66 [2026-10-22 11:08:46][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step711 [2026-10-22 11:08:46][INFO][training.pretrain_engine] Starting global step 712 (accumulating 32 micro-batches)... [2026-10-22 11:08:46][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 11:08:46][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-22 11:08:46][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-22 11:08:46][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-22 11:08:46][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step711 to Phase-Technologies/forge-500m-pretrain (path: phase2_step711)... [2026-10-22 11:08:46][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-22 11:08:55][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-22 11:08:55][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step711 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-22 11:09:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.0629 [2026-10-22 11:09:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6927 [2026-10-22 11:09:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3175 [2026-10-22 11:09:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3701 [2026-10-22 11:09:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4593 [2026-10-22 11:09:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4766 [2026-10-22 11:09:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2488 [2026-10-22 11:09:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7690 [2026-10-22 11:09:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9665 [2026-10-22 11:09:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1111 [2026-10-22 11:09:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2989 [2026-10-22 11:09:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4398 [2026-10-22 11:09:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3129 [2026-10-22 11:09:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=3.8814 [2026-10-22 11:09:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5216 [2026-10-22 11:09:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3256 [2026-10-22 11:10:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3918 [2026-10-22 11:10:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2502 [2026-10-22 11:10:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3424 [2026-10-22 11:10:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3724 [2026-10-22 11:10:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0535 [2026-10-22 11:10:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0112 [2026-10-22 11:10:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6322 [2026-10-22 11:10:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9589 [2026-10-22 11:10:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4545 [2026-10-22 11:10:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7338 [2026-10-22 11:10:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6400 [2026-10-22 11:10:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4201 [2026-10-22 11:11:03][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,612 | MFU: 1.8% [2026-10-22 11:11:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=3.7281 [2026-10-22 11:11:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.1851 [2026-10-22 11:11:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.1463 [2026-10-22 11:11:08][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2483 [2026-10-22 11:11:08][INFO][training.pretrain_engine] [Ph2|Step 712] loss=4.8791 | ppl=131.51 | aux=0.02505 | lr=2.97e-04 | gnorm=0.715 | tok/s=29,312 | MFU=7.0% | step=143089ms | tokens=1.549B | mem=4.2/32.7GB | cost=$58.42/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 18:39 (127.5h) | Phase3: SKIPPED | SFT: Oct 27 23:23 (132.2h) | DPO (Final): Oct 27 23:57 (132.8h) [2026-10-22 11:11:08][INFO][training.pretrain_engine] Starting global step 713 (accumulating 32 micro-batches)... [2026-10-22 11:11:08][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 11:11:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2057 [2026-10-22 11:11:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7464 [2026-10-22 11:11:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3894 [2026-10-22 11:11:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3361 [2026-10-22 11:11:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3053 [2026-10-22 11:11:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2632 [2026-10-22 11:11:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0411 [2026-10-22 11:11:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7664 [2026-10-22 11:12:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.0127 [2026-10-22 11:12:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7352 [2026-10-22 11:12:01][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7752 [2026-10-22 11:12:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2364 [2026-10-22 11:12:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6903 [2026-10-22 11:12:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1269 [2026-10-22 11:12:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1492 [2026-10-22 11:12:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8059 [2026-10-22 11:12:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5609 [2026-10-22 11:12:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9211 [2026-10-22 11:12:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6951 [2026-10-22 11:12:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8496 [2026-10-22 11:12:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1056 [2026-10-22 11:12:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6060 [2026-10-22 11:12:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0218 [2026-10-22 11:12:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6330 [2026-10-22 11:13:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6201 [2026-10-22 11:13:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8442 [2026-10-22 11:13:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5943 [2026-10-22 11:13:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4172 [2026-10-22 11:13:26][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,617 | MFU: 1.8% [2026-10-22 11:13:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0667 [2026-10-22 11:13:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3343 [2026-10-22 11:13:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1555 [2026-10-22 11:13:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3106 [2026-10-22 11:13:31][INFO][training.pretrain_engine] [Ph2|Step 713] loss=4.7865 | ppl=119.88 | aux=0.02506 | lr=2.97e-04 | gnorm=0.711 | tok/s=29,313 | MFU=7.0% | step=143089ms | tokens=1.550B | mem=4.2/32.7GB | cost=$58.50/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 18:41 (127.5h) | Phase3: SKIPPED | SFT: Oct 27 23:25 (132.2h) | DPO (Final): Oct 27 23:59 (132.8h) [2026-10-22 11:13:31][INFO][training.pretrain_engine] Starting global step 714 (accumulating 32 micro-batches)... [2026-10-22 11:13:31][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 11:13:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8801 [2026-10-22 11:13:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0054 [2026-10-22 11:13:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9146 [2026-10-22 11:13:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9999 [2026-10-22 11:14:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8033 [2026-10-22 11:14:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6434 [2026-10-22 11:14:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9456 [2026-10-22 11:14:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4016 [2026-10-22 11:14:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3480 [2026-10-22 11:14:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2008 [2026-10-22 11:14:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5405 [2026-10-22 11:14:24][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5872 [2026-10-22 11:14:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4862 [2026-10-22 11:14:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7845 [2026-10-22 11:14:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8991 [2026-10-22 11:14:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7299 [2026-10-22 11:14:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5930 [2026-10-22 11:15:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6956 [2026-10-22 11:15:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6531 [2026-10-22 11:15:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3641 [2026-10-22 11:15:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8376 [2026-10-22 11:15:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9627 [2026-10-22 11:15:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5907 [2026-10-22 11:15:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.1090 [2026-10-22 11:15:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4805 [2026-10-22 11:15:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2361 [2026-10-22 11:15:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3081 [2026-10-22 11:15:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4735 [2026-10-22 11:15:48][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,615 | MFU: 1.8% [2026-10-22 11:15:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1579 [2026-10-22 11:15:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.7291 [2026-10-22 11:15:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2791 [2026-10-22 11:15:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1863 [2026-10-22 11:15:54][INFO][training.pretrain_engine] [Ph2|Step 714] loss=4.7484 | ppl=115.39 | aux=0.02505 | lr=2.97e-04 | gnorm=0.719 | tok/s=29,314 | MFU=7.0% | step=143081ms | tokens=1.551B | mem=4.2/32.7GB | cost=$58.59/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 18:42 (127.4h) | Phase3: SKIPPED | SFT: Oct 27 23:26 (132.2h) | DPO (Final): Oct 28 00:00 (132.7h) [2026-10-22 11:15:54][INFO][training.pretrain_engine] Starting global step 715 (accumulating 32 micro-batches)... [2026-10-22 11:15:54][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 11:16:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6990 [2026-10-22 11:16:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0495 [2026-10-22 11:16:11][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0445 [2026-10-22 11:16:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9674 [2026-10-22 11:16:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2379 [2026-10-22 11:16:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3115 [2026-10-22 11:16:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5669 [2026-10-22 11:16:29][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1973 [2026-10-22 11:16:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9033 [2026-10-22 11:16:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2088 [2026-10-22 11:16:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0868 [2026-10-22 11:16:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6613 [2026-10-22 11:17:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9004 [2026-10-22 11:17:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7110 [2026-10-22 11:17:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3616 [2026-10-22 11:17:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2001 [2026-10-22 11:17:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7595 [2026-10-22 11:17:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8938 [2026-10-22 11:17:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6141 [2026-10-22 11:17:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5518 [2026-10-22 11:17:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2506 [2026-10-22 11:17:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7086 [2026-10-22 11:17:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7532 [2026-10-22 11:17:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2143 [2026-10-22 11:17:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=3.8632 [2026-10-22 11:17:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=3.8468 [2026-10-22 11:17:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7668 [2026-10-22 11:17:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7407 [2026-10-22 11:18:11][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,620 | MFU: 1.8% [2026-10-22 11:18:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4085 [2026-10-22 11:18:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4436 [2026-10-22 11:18:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7214 [2026-10-22 11:18:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4020 [2026-10-22 11:18:17][INFO][training.pretrain_engine] [Ph2|Step 715] loss=4.6435 | ppl=103.90 | aux=0.02505 | lr=2.97e-04 | gnorm=0.656 | tok/s=29,315 | MFU=7.0% | step=143078ms | tokens=1.552B | mem=4.2/32.7GB | cost=$58.67/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 18:44 (127.4h) | Phase3: SKIPPED | SFT: Oct 27 23:28 (132.2h) | DPO (Final): Oct 28 00:02 (132.7h) [2026-10-22 11:18:17][INFO][training.pretrain_engine] Starting global step 716 (accumulating 32 micro-batches)... [2026-10-22 11:18:17][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 11:18:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4933 [2026-10-22 11:18:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5374 [2026-10-22 11:18:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0283 [2026-10-22 11:18:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8138 [2026-10-22 11:18:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1821 [2026-10-22 11:18:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.9551 [2026-10-22 11:18:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3461 [2026-10-22 11:18:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1657 [2026-10-22 11:19:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7787 [2026-10-22 11:19:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7708 [2026-10-22 11:19:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8692 [2026-10-22 11:19:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8204 [2026-10-22 11:19:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2465 [2026-10-22 11:19:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7324 [2026-10-22 11:19:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7040 [2026-10-22 11:19:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0069 [2026-10-22 11:19:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1908 [2026-10-22 11:19:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0775 [2026-10-22 11:19:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8873 [2026-10-22 11:19:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1505 [2026-10-22 11:20:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8911 [2026-10-22 11:20:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8358 [2026-10-22 11:20:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4383 [2026-10-22 11:20:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6044 [2026-10-22 11:20:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0147 [2026-10-22 11:20:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5404 [2026-10-22 11:20:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6807 [2026-10-22 11:20:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7862 [2026-10-22 11:20:34][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,622 | MFU: 1.8% [2026-10-22 11:20:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2655 [2026-10-22 11:20:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6860 [2026-10-22 11:20:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7756 [2026-10-22 11:20:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.0672 [2026-10-22 11:20:40][INFO][training.pretrain_engine] [Ph2|Step 716] loss=4.6964 | ppl=109.56 | aux=0.02505 | lr=2.97e-04 | gnorm=0.598 | tok/s=29,316 | MFU=7.0% | step=143073ms | tokens=1.553B | mem=4.2/32.7GB | cost=$58.75/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 18:45 (127.4h) | Phase3: SKIPPED | SFT: Oct 27 23:29 (132.2h) | DPO (Final): Oct 28 00:03 (132.7h) [2026-10-22 11:20:40][INFO][training.pretrain_engine] Starting global step 717 (accumulating 32 micro-batches)... [2026-10-22 11:20:40][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 11:20:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4962 [2026-10-22 11:20:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8114 [2026-10-22 11:20:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8945 [2026-10-22 11:20:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7343 [2026-10-22 11:21:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5598 [2026-10-22 11:21:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7029 [2026-10-22 11:21:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4389 [2026-10-22 11:21:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3749 [2026-10-22 11:21:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9458 [2026-10-22 11:21:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1675 [2026-10-22 11:21:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4960 [2026-10-22 11:21:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6427 [2026-10-22 11:21:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6694 [2026-10-22 11:21:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5349 [2026-10-22 11:21:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8682 [2026-10-22 11:21:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7004 [2026-10-22 11:22:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5234 [2026-10-22 11:22:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7522 [2026-10-22 11:22:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4636 [2026-10-22 11:22:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4823 [2026-10-22 11:22:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8156 [2026-10-22 11:22:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3875 [2026-10-22 11:22:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4158 [2026-10-22 11:22:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7110 [2026-10-22 11:22:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.0911 [2026-10-22 11:22:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5641 [2026-10-22 11:22:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6032 [2026-10-22 11:22:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4826 [2026-10-22 11:22:57][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,634 | MFU: 1.8% [2026-10-22 11:23:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.1376 [2026-10-22 11:23:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4729 [2026-10-22 11:23:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0211 [2026-10-22 11:23:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6029 [2026-10-22 11:23:03][INFO][training.pretrain_engine] [Ph2|Step 717] loss=4.7781 | ppl=118.88 | aux=0.02505 | lr=2.96e-04 | gnorm=0.535 | tok/s=29,317 | MFU=7.0% | step=143068ms | tokens=1.554B | mem=4.2/32.7GB | cost=$58.83/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 18:47 (127.4h) | Phase3: SKIPPED | SFT: Oct 27 23:31 (132.1h) | DPO (Final): Oct 28 00:05 (132.7h) [2026-10-22 11:23:03][INFO][training.pretrain_engine] Starting global step 718 (accumulating 32 micro-batches)... [2026-10-22 11:23:03][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 11:23:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3868 [2026-10-22 11:23:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5645 [2026-10-22 11:23:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0855 [2026-10-22 11:23:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8243 [2026-10-22 11:23:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.6430 [2026-10-22 11:23:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.6406 [2026-10-22 11:23:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.9441 [2026-10-22 11:23:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1886 [2026-10-22 11:23:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7098 [2026-10-22 11:23:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5859 [2026-10-22 11:23:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8094 [2026-10-22 11:23:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9460 [2026-10-22 11:24:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9887 [2026-10-22 11:24:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1258 [2026-10-22 11:24:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8305 [2026-10-22 11:24:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9609 [2026-10-22 11:24:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7568 [2026-10-22 11:24:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6774 [2026-10-22 11:24:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5248 [2026-10-22 11:24:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1093 [2026-10-22 11:24:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8720 [2026-10-22 11:24:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8362 [2026-10-22 11:24:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7347 [2026-10-22 11:24:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2567 [2026-10-22 11:25:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4094 [2026-10-22 11:25:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7311 [2026-10-22 11:25:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7288 [2026-10-22 11:25:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7838 [2026-10-22 11:25:20][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,618 | MFU: 1.8% [2026-10-22 11:25:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6558 [2026-10-22 11:25:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5937 [2026-10-22 11:25:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6532 [2026-10-22 11:25:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6493 [2026-10-22 11:25:26][INFO][training.pretrain_engine] [Ph2|Step 718] loss=4.7654 | ppl=117.38 | aux=0.02504 | lr=2.96e-04 | gnorm=0.617 | tok/s=29,317 | MFU=7.0% | step=143065ms | tokens=1.555B | mem=4.2/32.7GB | cost=$58.91/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 18:48 (127.4h) | Phase3: SKIPPED | SFT: Oct 27 23:33 (132.1h) | DPO (Final): Oct 28 00:07 (132.7h) [2026-10-22 11:25:26][INFO][training.pretrain_engine] Starting global step 719 (accumulating 32 micro-batches)... [2026-10-22 11:25:26][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 11:25:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6705 [2026-10-22 11:25:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9111 [2026-10-22 11:25:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5897 [2026-10-22 11:25:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1018 [2026-10-22 11:26:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2607 [2026-10-22 11:26:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6382 [2026-10-22 11:26:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4388 [2026-10-22 11:26:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4350 [2026-10-22 11:26:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3442 [2026-10-22 11:26:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7765 [2026-10-22 11:26:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3858 [2026-10-22 11:26:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3617 [2026-10-22 11:26:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6232 [2026-10-22 11:26:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0174 [2026-10-22 11:26:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0633 [2026-10-22 11:26:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6886 [2026-10-22 11:26:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=3.6133 [2026-10-22 11:26:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=3.6472 [2026-10-22 11:26:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=3.4934 [2026-10-22 11:26:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3000 [2026-10-22 11:27:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6421 [2026-10-22 11:27:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9399 [2026-10-22 11:27:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0150 [2026-10-22 11:27:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8088 [2026-10-22 11:27:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4615 [2026-10-22 11:27:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8439 [2026-10-22 11:27:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9209 [2026-10-22 11:27:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7297 [2026-10-22 11:27:44][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,619 | MFU: 1.8% [2026-10-22 11:27:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9372 [2026-10-22 11:27:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0914 [2026-10-22 11:27:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5390 [2026-10-22 11:27:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0350 [2026-10-22 11:27:49][INFO][training.pretrain_engine] [Ph2|Step 719] loss=4.8232 | ppl=124.36 | aux=0.02505 | lr=2.96e-04 | gnorm=0.562 | tok/s=29,317 | MFU=7.0% | step=143065ms | tokens=1.556B | mem=4.2/32.7GB | cost=$59.00/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 18:50 (127.4h) | Phase3: SKIPPED | SFT: Oct 27 23:34 (132.1h) | DPO (Final): Oct 28 00:08 (132.7h) [2026-10-22 11:27:49][INFO][training.pretrain_engine] Starting global step 720 (accumulating 32 micro-batches)... [2026-10-22 11:27:49][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 11:28:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8052 [2026-10-22 11:28:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7312 [2026-10-22 11:28:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4209 [2026-10-22 11:28:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9193 [2026-10-22 11:28:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1312 [2026-10-22 11:28:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1097 [2026-10-22 11:28:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.0199 [2026-10-22 11:28:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7222 [2026-10-22 11:28:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3771 [2026-10-22 11:28:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3752 [2026-10-22 11:28:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8393 [2026-10-22 11:28:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1215 [2026-10-22 11:29:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6207 [2026-10-22 11:29:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3721 [2026-10-22 11:29:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6944 [2026-10-22 11:29:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3631 [2026-10-22 11:29:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7617 [2026-10-22 11:29:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5437 [2026-10-22 11:29:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3837 [2026-10-22 11:29:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3284 [2026-10-22 11:29:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6154 [2026-10-22 11:29:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5569 [2026-10-22 11:29:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9184 [2026-10-22 11:29:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0649 [2026-10-22 11:29:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5950 [2026-10-22 11:29:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9651 [2026-10-22 11:29:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9819 [2026-10-22 11:29:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6783 [2026-10-22 11:30:07][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,617 | MFU: 1.8% [2026-10-22 11:30:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8164 [2026-10-22 11:30:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7873 [2026-10-22 11:30:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8257 [2026-10-22 11:30:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7857 [2026-10-22 11:30:13][INFO][training.pretrain_engine] [Ph2|Step 720] loss=4.7405 | ppl=114.49 | aux=0.02505 | lr=2.96e-04 | gnorm=0.641 | tok/s=29,318 | MFU=7.0% | step=143061ms | tokens=1.557B | mem=4.2/32.7GB | cost=$59.08/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 18:52 (127.4h) | Phase3: SKIPPED | SFT: Oct 27 23:36 (132.1h) | DPO (Final): Oct 28 00:10 (132.7h) [2026-10-22 11:30:16][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step720 | Tokens: 1.557B | Spent: $59.08 | Remaining: $940.92 [2026-10-22 11:30:16][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step720 [2026-10-22 11:30:16][INFO][training.pretrain_engine] Starting global step 721 (accumulating 32 micro-batches)... [2026-10-22 11:30:16][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 11:30:16][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-22 11:30:16][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-22 11:30:16][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-22 11:30:16][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step720 to Phase-Technologies/forge-500m-pretrain (path: phase2_step720)... [2026-10-22 11:30:16][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-22 11:30:26][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-22 11:30:26][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step720 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-22 11:30:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8862 [2026-10-22 11:30:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9656 [2026-10-22 11:30:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9491 [2026-10-22 11:30:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8673 [2026-10-22 11:30:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9730 [2026-10-22 11:30:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9459 [2026-10-22 11:30:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7019 [2026-10-22 11:30:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8254 [2026-10-22 11:31:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=3.8380 [2026-10-22 11:31:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7006 [2026-10-22 11:31:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5767 [2026-10-22 11:31:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5599 [2026-10-22 11:31:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9144 [2026-10-22 11:31:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5897 [2026-10-22 11:31:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5936 [2026-10-22 11:31:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6280 [2026-10-22 11:31:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7685 [2026-10-22 11:31:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9341 [2026-10-22 11:31:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.2429 [2026-10-22 11:31:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7409 [2026-10-22 11:31:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1398 [2026-10-22 11:31:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5596 [2026-10-22 11:32:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9870 [2026-10-22 11:32:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0952 [2026-10-22 11:32:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.1320 [2026-10-22 11:32:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8180 [2026-10-22 11:32:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8863 [2026-10-22 11:32:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0009 [2026-10-22 11:32:34][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,580 | MFU: 1.8% [2026-10-22 11:32:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7191 [2026-10-22 11:32:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3726 [2026-10-22 11:32:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5974 [2026-10-22 11:32:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5090 [2026-10-22 11:32:39][INFO][training.pretrain_engine] [Ph2|Step 721] loss=4.7586 | ppl=116.59 | aux=0.02505 | lr=2.96e-04 | gnorm=0.598 | tok/s=29,319 | MFU=7.0% | step=143057ms | tokens=1.558B | mem=4.2/32.7GB | cost=$59.16/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 18:53 (127.4h) | Phase3: SKIPPED | SFT: Oct 27 23:37 (132.1h) | DPO (Final): Oct 28 00:12 (132.7h) [2026-10-22 11:32:39][INFO][training.pretrain_engine] Starting global step 722 (accumulating 32 micro-batches)... [2026-10-22 11:32:39][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 11:32:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0825 [2026-10-22 11:32:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6495 [2026-10-22 11:32:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4981 [2026-10-22 11:32:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9446 [2026-10-22 11:33:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0438 [2026-10-22 11:33:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2651 [2026-10-22 11:33:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4609 [2026-10-22 11:33:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0880 [2026-10-22 11:33:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8859 [2026-10-22 11:33:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2119 [2026-10-22 11:33:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1182 [2026-10-22 11:33:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9450 [2026-10-22 11:33:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6323 [2026-10-22 11:33:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2600 [2026-10-22 11:33:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6598 [2026-10-22 11:33:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1371 [2026-10-22 11:34:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3711 [2026-10-22 11:34:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3265 [2026-10-22 11:34:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8373 [2026-10-22 11:34:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9235 [2026-10-22 11:34:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2766 [2026-10-22 11:34:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2392 [2026-10-22 11:34:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8356 [2026-10-22 11:34:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1573 [2026-10-22 11:34:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.0079 [2026-10-22 11:34:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5656 [2026-10-22 11:34:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2047 [2026-10-22 11:34:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.1024 [2026-10-22 11:34:57][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,580 | MFU: 1.8% [2026-10-22 11:35:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6525 [2026-10-22 11:35:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6436 [2026-10-22 11:35:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6100 [2026-10-22 11:35:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8123 [2026-10-22 11:35:02][INFO][training.pretrain_engine] [Ph2|Step 722] loss=4.8214 | ppl=124.14 | aux=0.02505 | lr=2.96e-04 | gnorm=0.723 | tok/s=29,318 | MFU=7.0% | step=143064ms | tokens=1.559B | mem=4.2/32.7GB | cost=$59.24/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 18:55 (127.3h) | Phase3: SKIPPED | SFT: Oct 27 23:40 (132.1h) | DPO (Final): Oct 28 00:14 (132.7h) [2026-10-22 11:35:02][INFO][training.pretrain_engine] Starting global step 723 (accumulating 32 micro-batches)... [2026-10-22 11:35:02][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 11:35:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2986 [2026-10-22 11:35:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9070 [2026-10-22 11:35:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9339 [2026-10-22 11:35:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0602 [2026-10-22 11:35:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4931 [2026-10-22 11:35:37][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.3871 [2026-10-22 11:35:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9659 [2026-10-22 11:35:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0343 [2026-10-22 11:35:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4115 [2026-10-22 11:35:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0696 [2026-10-22 11:35:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.1467 [2026-10-22 11:35:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9853 [2026-10-22 11:36:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8121 [2026-10-22 11:36:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0604 [2026-10-22 11:36:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9113 [2026-10-22 11:36:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2791 [2026-10-22 11:36:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5672 [2026-10-22 11:36:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9780 [2026-10-22 11:36:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6859 [2026-10-22 11:36:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6469 [2026-10-22 11:36:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9220 [2026-10-22 11:36:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9064 [2026-10-22 11:36:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9212 [2026-10-22 11:36:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8698 [2026-10-22 11:37:06][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.0023 [2026-10-22 11:37:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7225 [2026-10-22 11:37:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3254 [2026-10-22 11:37:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6003 [2026-10-22 11:37:20][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,609 | MFU: 1.8% [2026-10-22 11:37:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2032 [2026-10-22 11:37:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9777 [2026-10-22 11:37:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8821 [2026-10-22 11:37:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0031 [2026-10-22 11:37:25][INFO][training.pretrain_engine] [Ph2|Step 723] loss=4.9108 | ppl=135.74 | aux=0.02505 | lr=2.96e-04 | gnorm=0.715 | tok/s=29,319 | MFU=7.0% | step=143056ms | tokens=1.560B | mem=4.2/32.7GB | cost=$59.32/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 18:57 (127.3h) | Phase3: SKIPPED | SFT: Oct 27 23:41 (132.1h) | DPO (Final): Oct 28 00:15 (132.6h) [2026-10-22 11:37:25][INFO][training.pretrain_engine] Starting global step 724 (accumulating 32 micro-batches)... [2026-10-22 11:37:25][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 11:37:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5072 [2026-10-22 11:37:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8500 [2026-10-22 11:37:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7756 [2026-10-22 11:37:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5765 [2026-10-22 11:38:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4317 [2026-10-22 11:38:00][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2624 [2026-10-22 11:38:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1738 [2026-10-22 11:38:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9250 [2026-10-22 11:38:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6212 [2026-10-22 11:38:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5888 [2026-10-22 11:38:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0542 [2026-10-22 11:38:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2523 [2026-10-22 11:38:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8393 [2026-10-22 11:38:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=3.6924 [2026-10-22 11:38:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5540 [2026-10-22 11:38:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4755 [2026-10-22 11:38:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1224 [2026-10-22 11:38:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0002 [2026-10-22 11:38:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8674 [2026-10-22 11:38:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5877 [2026-10-22 11:39:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1185 [2026-10-22 11:39:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9759 [2026-10-22 11:39:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9106 [2026-10-22 11:39:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3414 [2026-10-22 11:39:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3474 [2026-10-22 11:39:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8589 [2026-10-22 11:39:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2478 [2026-10-22 11:39:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6031 [2026-10-22 11:39:43][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,616 | MFU: 1.8% [2026-10-22 11:39:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5015 [2026-10-22 11:39:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1043 [2026-10-22 11:39:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7369 [2026-10-22 11:39:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8935 [2026-10-22 11:39:48][INFO][training.pretrain_engine] [Ph2|Step 724] loss=4.7659 | ppl=117.44 | aux=0.02506 | lr=2.96e-04 | gnorm=0.715 | tok/s=29,321 | MFU=7.0% | step=143050ms | tokens=1.561B | mem=4.2/32.7GB | cost=$59.41/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 18:58 (127.3h) | Phase3: SKIPPED | SFT: Oct 27 23:42 (132.1h) | DPO (Final): Oct 28 00:17 (132.6h) [2026-10-22 11:39:48][INFO][training.pretrain_engine] Starting global step 725 (accumulating 32 micro-batches)... [2026-10-22 11:39:48][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 11:40:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6408 [2026-10-22 11:40:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7367 [2026-10-22 11:40:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5551 [2026-10-22 11:40:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7487 [2026-10-22 11:40:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8804 [2026-10-22 11:40:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2536 [2026-10-22 11:40:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1410 [2026-10-22 11:40:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9800 [2026-10-22 11:40:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6699 [2026-10-22 11:40:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5577 [2026-10-22 11:40:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2748 [2026-10-22 11:40:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7980 [2026-10-22 11:40:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.1530 [2026-10-22 11:40:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7887 [2026-10-22 11:40:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5430 [2026-10-22 11:41:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6960 [2026-10-22 11:41:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5917 [2026-10-22 11:41:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0196 [2026-10-22 11:41:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9417 [2026-10-22 11:41:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2078 [2026-10-22 11:41:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7627 [2026-10-22 11:41:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7601 [2026-10-22 11:41:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9484 [2026-10-22 11:41:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3799 [2026-10-22 11:41:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9627 [2026-10-22 11:41:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2649 [2026-10-22 11:41:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0378 [2026-10-22 11:41:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6928 [2026-10-22 11:42:06][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,615 | MFU: 1.8% [2026-10-22 11:42:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5529 [2026-10-22 11:42:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8816 [2026-10-22 11:42:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7502 [2026-10-22 11:42:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0397 [2026-10-22 11:42:11][INFO][training.pretrain_engine] [Ph2|Step 725] loss=4.7980 | ppl=121.26 | aux=0.02505 | lr=2.96e-04 | gnorm=0.637 | tok/s=29,321 | MFU=7.0% | step=143048ms | tokens=1.562B | mem=4.2/32.7GB | cost=$59.49/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 19:00 (127.3h) | Phase3: SKIPPED | SFT: Oct 27 23:44 (132.0h) | DPO (Final): Oct 28 00:18 (132.6h) [2026-10-22 11:42:11][INFO][training.pretrain_engine] Starting global step 726 (accumulating 32 micro-batches)... [2026-10-22 11:42:11][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 11:42:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3752 [2026-10-22 11:42:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2041 [2026-10-22 11:42:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9039 [2026-10-22 11:42:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9297 [2026-10-22 11:42:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0732 [2026-10-22 11:42:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5070 [2026-10-22 11:42:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0504 [2026-10-22 11:42:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3268 [2026-10-22 11:43:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6129 [2026-10-22 11:43:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=3.8486 [2026-10-22 11:43:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.1289 [2026-10-22 11:43:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.0111 [2026-10-22 11:43:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8784 [2026-10-22 11:43:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7929 [2026-10-22 11:43:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0007 [2026-10-22 11:43:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6509 [2026-10-22 11:43:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9443 [2026-10-22 11:43:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7264 [2026-10-22 11:43:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8766 [2026-10-22 11:43:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1929 [2026-10-22 11:43:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0994 [2026-10-22 11:43:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9193 [2026-10-22 11:43:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6173 [2026-10-22 11:43:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1190 [2026-10-22 11:44:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5717 [2026-10-22 11:44:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4141 [2026-10-22 11:44:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4918 [2026-10-22 11:44:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4089 [2026-10-22 11:44:29][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,617 | MFU: 1.8% [2026-10-22 11:44:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6657 [2026-10-22 11:44:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3600 [2026-10-22 11:44:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5664 [2026-10-22 11:44:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5084 [2026-10-22 11:44:34][INFO][training.pretrain_engine] [Ph2|Step 726] loss=4.7639 | ppl=117.20 | aux=0.02505 | lr=2.96e-04 | gnorm=0.621 | tok/s=29,323 | MFU=7.0% | step=143039ms | tokens=1.563B | mem=4.2/32.7GB | cost=$59.57/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 19:01 (127.3h) | Phase3: SKIPPED | SFT: Oct 27 23:45 (132.0h) | DPO (Final): Oct 28 00:20 (132.6h) [2026-10-22 11:44:34][INFO][training.pretrain_engine] Starting global step 727 (accumulating 32 micro-batches)... [2026-10-22 11:44:34][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 11:44:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3562 [2026-10-22 11:44:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5585 [2026-10-22 11:44:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.9256 [2026-10-22 11:44:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6759 [2026-10-22 11:45:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7562 [2026-10-22 11:45:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3305 [2026-10-22 11:45:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9060 [2026-10-22 11:45:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9915 [2026-10-22 11:45:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5407 [2026-10-22 11:45:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9157 [2026-10-22 11:45:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7155 [2026-10-22 11:45:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7775 [2026-10-22 11:45:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8597 [2026-10-22 11:45:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.1179 [2026-10-22 11:45:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4697 [2026-10-22 11:45:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7909 [2026-10-22 11:46:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9099 [2026-10-22 11:46:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9715 [2026-10-22 11:46:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8764 [2026-10-22 11:46:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1539 [2026-10-22 11:46:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=3.8910 [2026-10-22 11:46:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7303 [2026-10-22 11:46:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9887 [2026-10-22 11:46:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4767 [2026-10-22 11:46:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7626 [2026-10-22 11:46:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8650 [2026-10-22 11:46:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9827 [2026-10-22 11:46:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8706 [2026-10-22 11:46:52][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,606 | MFU: 1.8% [2026-10-22 11:46:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6220 [2026-10-22 11:46:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.0912 [2026-10-22 11:46:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3603 [2026-10-22 11:46:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=3.3156 [2026-10-22 11:46:57][INFO][training.pretrain_engine] [Ph2|Step 727] loss=4.8328 | ppl=125.56 | aux=0.02505 | lr=2.96e-04 | gnorm=0.605 | tok/s=29,324 | MFU=7.0% | step=143033ms | tokens=1.564B | mem=4.2/32.7GB | cost=$59.65/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 19:03 (127.3h) | Phase3: SKIPPED | SFT: Oct 27 23:47 (132.0h) | DPO (Final): Oct 28 00:21 (132.6h) [2026-10-22 11:46:57][INFO][training.pretrain_engine] Starting global step 728 (accumulating 32 micro-batches)... [2026-10-22 11:46:57][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 11:47:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6134 [2026-10-22 11:47:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7410 [2026-10-22 11:47:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5700 [2026-10-22 11:47:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6196 [2026-10-22 11:47:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1689 [2026-10-22 11:47:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3097 [2026-10-22 11:47:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.0206 [2026-10-22 11:47:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3547 [2026-10-22 11:47:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8527 [2026-10-22 11:47:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9704 [2026-10-22 11:47:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8829 [2026-10-22 11:47:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0172 [2026-10-22 11:48:07][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8976 [2026-10-22 11:48:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.1861 [2026-10-22 11:48:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8854 [2026-10-22 11:48:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4826 [2026-10-22 11:48:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4712 [2026-10-22 11:48:25][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6362 [2026-10-22 11:48:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4642 [2026-10-22 11:48:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5308 [2026-10-22 11:48:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9431 [2026-10-22 11:48:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8857 [2026-10-22 11:48:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2696 [2026-10-22 11:48:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9623 [2026-10-22 11:49:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6324 [2026-10-22 11:49:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5902 [2026-10-22 11:49:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0271 [2026-10-22 11:49:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7571 [2026-10-22 11:49:14][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,615 | MFU: 1.8% [2026-10-22 11:49:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0344 [2026-10-22 11:49:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1543 [2026-10-22 11:49:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9438 [2026-10-22 11:49:19][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1829 [2026-10-22 11:49:19][INFO][training.pretrain_engine] [Ph2|Step 728] loss=4.7250 | ppl=112.73 | aux=0.02505 | lr=2.96e-04 | gnorm=0.543 | tok/s=29,325 | MFU=7.0% | step=143027ms | tokens=1.566B | mem=4.2/32.7GB | cost=$59.73/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 19:04 (127.3h) | Phase3: SKIPPED | SFT: Oct 27 23:48 (132.0h) | DPO (Final): Oct 28 00:22 (132.6h) [2026-10-22 11:49:19][INFO][training.pretrain_engine] Starting global step 729 (accumulating 32 micro-batches)... [2026-10-22 11:49:19][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 11:49:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5061 [2026-10-22 11:49:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8735 [2026-10-22 11:49:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1529 [2026-10-22 11:49:37][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8940 [2026-10-22 11:49:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4462 [2026-10-22 11:49:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.7425 [2026-10-22 11:49:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3962 [2026-10-22 11:49:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4743 [2026-10-22 11:50:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7798 [2026-10-22 11:50:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1311 [2026-10-22 11:50:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1651 [2026-10-22 11:50:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4632 [2026-10-22 11:50:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4103 [2026-10-22 11:50:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1124 [2026-10-22 11:50:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2552 [2026-10-22 11:50:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6595 [2026-10-22 11:50:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4007 [2026-10-22 11:50:48][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6971 [2026-10-22 11:50:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.0637 [2026-10-22 11:50:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7732 [2026-10-22 11:51:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6112 [2026-10-22 11:51:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5438 [2026-10-22 11:51:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6406 [2026-10-22 11:51:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5991 [2026-10-22 11:51:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2445 [2026-10-22 11:51:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3057 [2026-10-22 11:51:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0448 [2026-10-22 11:51:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.5510 [2026-10-22 11:51:37][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,606 | MFU: 1.8% [2026-10-22 11:51:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9623 [2026-10-22 11:51:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6784 [2026-10-22 11:51:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7981 [2026-10-22 11:51:42][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6446 [2026-10-22 11:51:42][INFO][training.pretrain_engine] [Ph2|Step 729] loss=4.7752 | ppl=118.53 | aux=0.02505 | lr=2.96e-04 | gnorm=0.516 | tok/s=29,327 | MFU=7.0% | step=143020ms | tokens=1.567B | mem=4.2/32.7GB | cost=$59.81/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 19:06 (127.2h) | Phase3: SKIPPED | SFT: Oct 27 23:50 (132.0h) | DPO (Final): Oct 28 00:24 (132.5h) [2026-10-22 11:51:45][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step729 | Tokens: 1.567B | Spent: $59.82 | Remaining: $940.18 [2026-10-22 11:51:45][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step729 [2026-10-22 11:51:45][INFO][training.pretrain_engine] Starting global step 730 (accumulating 32 micro-batches)... [2026-10-22 11:51:45][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 11:51:45][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-22 11:51:45][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-22 11:51:45][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-22 11:51:45][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step729 to Phase-Technologies/forge-500m-pretrain (path: phase2_step729)... [2026-10-22 11:51:45][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-22 11:51:55][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-22 11:51:55][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step729 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-22 11:52:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8088 [2026-10-22 11:52:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4677 [2026-10-22 11:52:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3149 [2026-10-22 11:52:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6617 [2026-10-22 11:52:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9997 [2026-10-22 11:52:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8541 [2026-10-22 11:52:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8642 [2026-10-22 11:52:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7863 [2026-10-22 11:52:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1657 [2026-10-22 11:52:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1000 [2026-10-22 11:52:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2578 [2026-10-22 11:52:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1657 [2026-10-22 11:52:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5011 [2026-10-22 11:52:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8264 [2026-10-22 11:52:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9200 [2026-10-22 11:52:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8205 [2026-10-22 11:53:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7609 [2026-10-22 11:53:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.1992 [2026-10-22 11:53:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5187 [2026-10-22 11:53:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8039 [2026-10-22 11:53:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.1204 [2026-10-22 11:53:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5521 [2026-10-22 11:53:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7923 [2026-10-22 11:53:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2396 [2026-10-22 11:53:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8138 [2026-10-22 11:53:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9416 [2026-10-22 11:53:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8616 [2026-10-22 11:53:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4542 [2026-10-22 11:54:03][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,624 | MFU: 1.8% [2026-10-22 11:54:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5572 [2026-10-22 11:54:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8510 [2026-10-22 11:54:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1095 [2026-10-22 11:54:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0333 [2026-10-22 11:54:07][INFO][training.pretrain_engine] [Ph2|Step 730] loss=4.7675 | ppl=117.62 | aux=0.02506 | lr=2.96e-04 | gnorm=0.523 | tok/s=29,330 | MFU=7.0% | step=143004ms | tokens=1.568B | mem=4.2/32.7GB | cost=$59.90/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 19:07 (127.2h) | Phase3: SKIPPED | SFT: Oct 27 23:51 (132.0h) | DPO (Final): Oct 28 00:25 (132.5h) [2026-10-22 11:54:07][INFO][training.pretrain_engine] Starting global step 731 (accumulating 32 micro-batches)... [2026-10-22 11:54:07][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 11:54:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7923 [2026-10-22 11:54:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0384 [2026-10-22 11:54:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0342 [2026-10-22 11:54:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1796 [2026-10-22 11:54:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7788 [2026-10-22 11:54:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8997 [2026-10-22 11:54:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7929 [2026-10-22 11:54:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8954 [2026-10-22 11:55:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.0343 [2026-10-22 11:55:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.1226 [2026-10-22 11:55:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=3.6897 [2026-10-22 11:55:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.0944 [2026-10-22 11:55:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1238 [2026-10-22 11:55:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4294 [2026-10-22 11:55:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7660 [2026-10-22 11:55:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7611 [2026-10-22 11:55:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1268 [2026-10-22 11:55:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8928 [2026-10-22 11:55:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1979 [2026-10-22 11:55:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1375 [2026-10-22 11:55:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9398 [2026-10-22 11:55:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7270 [2026-10-22 11:55:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6832 [2026-10-22 11:55:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0139 [2026-10-22 11:56:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6220 [2026-10-22 11:56:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3215 [2026-10-22 11:56:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3458 [2026-10-22 11:56:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.0017 [2026-10-22 11:56:25][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,618 | MFU: 1.8% [2026-10-22 11:56:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6870 [2026-10-22 11:56:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9063 [2026-10-22 11:56:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6089 [2026-10-22 11:56:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7886 [2026-10-22 11:56:30][INFO][training.pretrain_engine] [Ph2|Step 731] loss=4.7374 | ppl=114.14 | aux=0.02505 | lr=2.96e-04 | gnorm=0.516 | tok/s=29,330 | MFU=7.0% | step=143006ms | tokens=1.569B | mem=4.2/32.7GB | cost=$59.98/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 19:08 (127.2h) | Phase3: SKIPPED | SFT: Oct 27 23:53 (131.9h) | DPO (Final): Oct 28 00:27 (132.5h) [2026-10-22 11:56:30][INFO][training.pretrain_engine] Starting global step 732 (accumulating 32 micro-batches)... [2026-10-22 11:56:30][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 11:56:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0474 [2026-10-22 11:56:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0784 [2026-10-22 11:56:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8743 [2026-10-22 11:56:48][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8398 [2026-10-22 11:57:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4053 [2026-10-22 11:57:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.9283 [2026-10-22 11:57:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.8874 [2026-10-22 11:57:06][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.8835 [2026-10-22 11:57:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0677 [2026-10-22 11:57:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7764 [2026-10-22 11:57:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1199 [2026-10-22 11:57:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7535 [2026-10-22 11:57:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7092 [2026-10-22 11:57:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7348 [2026-10-22 11:57:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0017 [2026-10-22 11:57:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5548 [2026-10-22 11:57:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.2862 [2026-10-22 11:57:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5440 [2026-10-22 11:57:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7474 [2026-10-22 11:57:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7449 [2026-10-22 11:58:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2863 [2026-10-22 11:58:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2018 [2026-10-22 11:58:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5190 [2026-10-22 11:58:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4908 [2026-10-22 11:58:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7820 [2026-10-22 11:58:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.1427 [2026-10-22 11:58:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5907 [2026-10-22 11:58:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3916 [2026-10-22 11:58:47][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,616 | MFU: 1.8% [2026-10-22 11:58:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6579 [2026-10-22 11:58:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2335 [2026-10-22 11:58:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7373 [2026-10-22 11:58:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7781 [2026-10-22 11:58:52][INFO][training.pretrain_engine] [Ph2|Step 732] loss=4.7597 | ppl=116.71 | aux=0.02505 | lr=2.96e-04 | gnorm=0.555 | tok/s=29,331 | MFU=7.0% | step=142997ms | tokens=1.570B | mem=4.2/32.7GB | cost=$60.06/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 19:10 (127.2h) | Phase3: SKIPPED | SFT: Oct 27 23:54 (131.9h) | DPO (Final): Oct 28 00:28 (132.5h) [2026-10-22 11:58:52][INFO][training.pretrain_engine] Starting global step 733 (accumulating 32 micro-batches)... [2026-10-22 11:58:52][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 11:59:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4534 [2026-10-22 11:59:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6152 [2026-10-22 11:59:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4099 [2026-10-22 11:59:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4715 [2026-10-22 11:59:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5339 [2026-10-22 11:59:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7640 [2026-10-22 11:59:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1299 [2026-10-22 11:59:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4632 [2026-10-22 11:59:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5271 [2026-10-22 11:59:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2033 [2026-10-22 11:59:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3962 [2026-10-22 11:59:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5587 [2026-10-22 12:00:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5246 [2026-10-22 12:00:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6003 [2026-10-22 12:00:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3145 [2026-10-22 12:00:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7051 [2026-10-22 12:00:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7933 [2026-10-22 12:00:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9183 [2026-10-22 12:00:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8618 [2026-10-22 12:00:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1297 [2026-10-22 12:00:38][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1665 [2026-10-22 12:00:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6933 [2026-10-22 12:00:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5401 [2026-10-22 12:00:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8397 [2026-10-22 12:00:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9239 [2026-10-22 12:00:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0681 [2026-10-22 12:00:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1660 [2026-10-22 12:00:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6185 [2026-10-22 12:01:10][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,618 | MFU: 1.8% [2026-10-22 12:01:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5325 [2026-10-22 12:01:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7477 [2026-10-22 12:01:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5453 [2026-10-22 12:01:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2546 [2026-10-22 12:01:15][INFO][training.pretrain_engine] [Ph2|Step 733] loss=4.7307 | ppl=113.37 | aux=0.02505 | lr=2.96e-04 | gnorm=0.492 | tok/s=29,334 | MFU=7.0% | step=142986ms | tokens=1.571B | mem=4.2/32.7GB | cost=$60.14/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 19:11 (127.2h) | Phase3: SKIPPED | SFT: Oct 27 23:55 (131.9h) | DPO (Final): Oct 28 00:29 (132.5h) [2026-10-22 12:01:15][INFO][training.pretrain_engine] Starting global step 734 (accumulating 32 micro-batches)... [2026-10-22 12:01:15][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 12:01:32][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7673 [2026-10-22 12:01:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2955 [2026-10-22 12:01:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6545 [2026-10-22 12:01:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3394 [2026-10-22 12:01:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4784 [2026-10-22 12:01:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6390 [2026-10-22 12:01:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0396 [2026-10-22 12:01:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6871 [2026-10-22 12:02:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8843 [2026-10-22 12:02:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5732 [2026-10-22 12:02:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9683 [2026-10-22 12:02:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8638 [2026-10-22 12:02:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1766 [2026-10-22 12:02:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0080 [2026-10-22 12:02:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0737 [2026-10-22 12:02:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6654 [2026-10-22 12:02:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1243 [2026-10-22 12:02:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7282 [2026-10-22 12:02:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8120 [2026-10-22 12:02:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9142 [2026-10-22 12:03:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.1540 [2026-10-22 12:03:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6315 [2026-10-22 12:03:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3350 [2026-10-22 12:03:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.1741 [2026-10-22 12:03:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3751 [2026-10-22 12:03:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9983 [2026-10-22 12:03:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2517 [2026-10-22 12:03:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2114 [2026-10-22 12:03:33][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,600 | MFU: 1.8% [2026-10-22 12:03:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.0034 [2026-10-22 12:03:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7018 [2026-10-22 12:03:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4133 [2026-10-22 12:03:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2770 [2026-10-22 12:03:38][INFO][training.pretrain_engine] [Ph2|Step 734] loss=4.6639 | ppl=106.05 | aux=0.02505 | lr=2.96e-04 | gnorm=0.494 | tok/s=29,335 | MFU=7.0% | step=142977ms | tokens=1.572B | mem=4.2/32.7GB | cost=$60.22/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 19:12 (127.2h) | Phase3: SKIPPED | SFT: Oct 27 23:56 (131.9h) | DPO (Final): Oct 28 00:30 (132.5h) [2026-10-22 12:03:38][INFO][training.pretrain_engine] Starting global step 735 (accumulating 32 micro-batches)... [2026-10-22 12:03:38][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 12:03:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6104 [2026-10-22 12:03:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8342 [2026-10-22 12:03:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.1860 [2026-10-22 12:03:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4155 [2026-10-22 12:04:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9584 [2026-10-22 12:04:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4867 [2026-10-22 12:04:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0683 [2026-10-22 12:04:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9172 [2026-10-22 12:04:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6137 [2026-10-22 12:04:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.0084 [2026-10-22 12:04:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.1501 [2026-10-22 12:04:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=3.6936 [2026-10-22 12:04:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1408 [2026-10-22 12:04:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9809 [2026-10-22 12:04:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1269 [2026-10-22 12:04:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8365 [2026-10-22 12:05:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8085 [2026-10-22 12:05:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5558 [2026-10-22 12:05:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5546 [2026-10-22 12:05:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.2425 [2026-10-22 12:05:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7984 [2026-10-22 12:05:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4726 [2026-10-22 12:05:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7360 [2026-10-22 12:05:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6298 [2026-10-22 12:05:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2539 [2026-10-22 12:05:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9864 [2026-10-22 12:05:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0429 [2026-10-22 12:05:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9681 [2026-10-22 12:05:56][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,593 | MFU: 1.8% [2026-10-22 12:06:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5375 [2026-10-22 12:06:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.1917 [2026-10-22 12:06:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3532 [2026-10-22 12:06:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5277 [2026-10-22 12:06:00][INFO][training.pretrain_engine] [Ph2|Step 735] loss=4.6150 | ppl=100.99 | aux=0.02505 | lr=2.96e-04 | gnorm=0.570 | tok/s=29,337 | MFU=7.0% | step=142968ms | tokens=1.573B | mem=4.2/32.7GB | cost=$60.31/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 19:14 (127.1h) | Phase3: SKIPPED | SFT: Oct 27 23:58 (131.9h) | DPO (Final): Oct 28 00:32 (132.4h) [2026-10-22 12:06:00][INFO][training.pretrain_engine] Starting global step 736 (accumulating 32 micro-batches)... [2026-10-22 12:06:00][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 12:06:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7451 [2026-10-22 12:06:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4861 [2026-10-22 12:06:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.6845 [2026-10-22 12:06:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7889 [2026-10-22 12:06:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3815 [2026-10-22 12:06:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3850 [2026-10-22 12:06:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4832 [2026-10-22 12:06:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4853 [2026-10-22 12:06:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9808 [2026-10-22 12:06:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7235 [2026-10-22 12:06:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9533 [2026-10-22 12:06:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6054 [2026-10-22 12:07:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9503 [2026-10-22 12:07:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7566 [2026-10-22 12:07:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8377 [2026-10-22 12:07:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8609 [2026-10-22 12:07:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6307 [2026-10-22 12:07:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6893 [2026-10-22 12:07:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3057 [2026-10-22 12:07:30][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7630 [2026-10-22 12:07:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3155 [2026-10-22 12:07:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5335 [2026-10-22 12:07:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6933 [2026-10-22 12:07:48][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7892 [2026-10-22 12:08:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0343 [2026-10-22 12:08:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7726 [2026-10-22 12:08:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5433 [2026-10-22 12:08:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8156 [2026-10-22 12:08:19][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,560 | MFU: 1.8% [2026-10-22 12:08:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1864 [2026-10-22 12:08:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2428 [2026-10-22 12:08:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7834 [2026-10-22 12:08:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8520 [2026-10-22 12:08:23][INFO][training.pretrain_engine] [Ph2|Step 736] loss=4.6682 | ppl=106.51 | aux=0.02506 | lr=2.96e-04 | gnorm=0.535 | tok/s=29,338 | MFU=7.0% | step=142965ms | tokens=1.574B | mem=4.2/32.7GB | cost=$60.39/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 19:15 (127.1h) | Phase3: SKIPPED | SFT: Oct 27 23:59 (131.9h) | DPO (Final): Oct 28 00:33 (132.4h) [2026-10-22 12:08:23][INFO][training.pretrain_engine] Starting global step 737 (accumulating 32 micro-batches)... [2026-10-22 12:08:23][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 12:08:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8019 [2026-10-22 12:08:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7989 [2026-10-22 12:08:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8797 [2026-10-22 12:08:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8345 [2026-10-22 12:08:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0228 [2026-10-22 12:08:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5920 [2026-10-22 12:08:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7650 [2026-10-22 12:08:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4194 [2026-10-22 12:09:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8175 [2026-10-22 12:09:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5514 [2026-10-22 12:09:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1946 [2026-10-22 12:09:17][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8635 [2026-10-22 12:09:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0060 [2026-10-22 12:09:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0169 [2026-10-22 12:09:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9536 [2026-10-22 12:09:35][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8793 [2026-10-22 12:09:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3733 [2026-10-22 12:09:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5371 [2026-10-22 12:09:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6691 [2026-10-22 12:09:53][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4309 [2026-10-22 12:10:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3250 [2026-10-22 12:10:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4223 [2026-10-22 12:10:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0445 [2026-10-22 12:10:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6790 [2026-10-22 12:10:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9927 [2026-10-22 12:10:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0979 [2026-10-22 12:10:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2212 [2026-10-22 12:10:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9062 [2026-10-22 12:10:42][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,587 | MFU: 1.8% [2026-10-22 12:10:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4666 [2026-10-22 12:10:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2790 [2026-10-22 12:10:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1871 [2026-10-22 12:10:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0110 [2026-10-22 12:10:46][INFO][training.pretrain_engine] [Ph2|Step 737] loss=4.7585 | ppl=116.57 | aux=0.02506 | lr=2.96e-04 | gnorm=0.566 | tok/s=29,339 | MFU=7.0% | step=142959ms | tokens=1.575B | mem=4.2/32.7GB | cost=$60.47/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 19:17 (127.1h) | Phase3: SKIPPED | SFT: Oct 28 00:01 (131.8h) | DPO (Final): Oct 28 00:35 (132.4h) [2026-10-22 12:10:46][INFO][training.pretrain_engine] Starting global step 738 (accumulating 32 micro-batches)... [2026-10-22 12:10:46][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 12:11:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9789 [2026-10-22 12:11:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9482 [2026-10-22 12:11:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1701 [2026-10-22 12:11:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7498 [2026-10-22 12:11:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8743 [2026-10-22 12:11:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7548 [2026-10-22 12:11:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9500 [2026-10-22 12:11:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0562 [2026-10-22 12:11:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5084 [2026-10-22 12:11:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9502 [2026-10-22 12:11:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5987 [2026-10-22 12:11:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9626 [2026-10-22 12:11:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0245 [2026-10-22 12:11:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.9186 [2026-10-22 12:11:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2885 [2026-10-22 12:11:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0090 [2026-10-22 12:12:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8568 [2026-10-22 12:12:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6506 [2026-10-22 12:12:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8558 [2026-10-22 12:12:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6728 [2026-10-22 12:12:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.1562 [2026-10-22 12:12:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2436 [2026-10-22 12:12:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4753 [2026-10-22 12:12:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2228 [2026-10-22 12:12:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6294 [2026-10-22 12:12:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.1795 [2026-10-22 12:12:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2880 [2026-10-22 12:12:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4905 [2026-10-22 12:13:04][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,616 | MFU: 1.8% [2026-10-22 12:13:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7249 [2026-10-22 12:13:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7706 [2026-10-22 12:13:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5045 [2026-10-22 12:13:10][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8017 [2026-10-22 12:13:10][INFO][training.pretrain_engine] [Ph2|Step 738] loss=4.7205 | ppl=112.22 | aux=0.02506 | lr=2.96e-04 | gnorm=0.617 | tok/s=29,340 | MFU=7.0% | step=142953ms | tokens=1.576B | mem=4.2/32.7GB | cost=$60.55/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 19:18 (127.1h) | Phase3: SKIPPED | SFT: Oct 28 00:02 (131.8h) | DPO (Final): Oct 28 00:36 (132.4h) [2026-10-22 12:13:13][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step738 | Tokens: 1.576B | Spent: $60.55 | Remaining: $939.45 [2026-10-22 12:13:13][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step738 [2026-10-22 12:13:13][INFO][training.pretrain_engine] Starting global step 739 (accumulating 32 micro-batches)... [2026-10-22 12:13:13][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 12:13:13][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-22 12:13:13][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-22 12:13:13][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-22 12:13:13][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step738 to Phase-Technologies/forge-500m-pretrain (path: phase2_step738)... [2026-10-22 12:13:13][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-22 12:13:23][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-22 12:13:23][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step738 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-22 12:13:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6002 [2026-10-22 12:13:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7245 [2026-10-22 12:13:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6425 [2026-10-22 12:13:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.9172 [2026-10-22 12:13:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4829 [2026-10-22 12:13:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7275 [2026-10-22 12:13:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3860 [2026-10-22 12:13:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6423 [2026-10-22 12:14:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5574 [2026-10-22 12:14:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3512 [2026-10-22 12:14:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3361 [2026-10-22 12:14:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=3.9433 [2026-10-22 12:14:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7460 [2026-10-22 12:14:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8096 [2026-10-22 12:14:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0722 [2026-10-22 12:14:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7524 [2026-10-22 12:14:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5386 [2026-10-22 12:14:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7117 [2026-10-22 12:14:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8193 [2026-10-22 12:14:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4022 [2026-10-22 12:14:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2996 [2026-10-22 12:14:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=3.8926 [2026-10-22 12:14:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2899 [2026-10-22 12:15:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3447 [2026-10-22 12:15:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8441 [2026-10-22 12:15:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5812 [2026-10-22 12:15:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5384 [2026-10-22 12:15:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3010 [2026-10-22 12:15:31][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,607 | MFU: 1.8% [2026-10-22 12:15:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9597 [2026-10-22 12:15:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6283 [2026-10-22 12:15:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7864 [2026-10-22 12:15:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6580 [2026-10-22 12:15:35][INFO][training.pretrain_engine] [Ph2|Step 739] loss=4.7275 | ppl=113.01 | aux=0.02507 | lr=2.96e-04 | gnorm=0.695 | tok/s=29,344 | MFU=7.0% | step=142936ms | tokens=1.577B | mem=4.2/32.7GB | cost=$60.64/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 19:19 (127.1h) | Phase3: SKIPPED | SFT: Oct 28 00:03 (131.8h) | DPO (Final): Oct 28 00:37 (132.4h) [2026-10-22 12:15:35][INFO][training.pretrain_engine] Starting global step 740 (accumulating 32 micro-batches)... [2026-10-22 12:15:35][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 12:15:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3095 [2026-10-22 12:15:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.6932 [2026-10-22 12:15:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2670 [2026-10-22 12:15:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.0894 [2026-10-22 12:16:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8726 [2026-10-22 12:16:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0007 [2026-10-22 12:16:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8452 [2026-10-22 12:16:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4553 [2026-10-22 12:16:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9695 [2026-10-22 12:16:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6423 [2026-10-22 12:16:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8574 [2026-10-22 12:16:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9410 [2026-10-22 12:16:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3946 [2026-10-22 12:16:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6136 [2026-10-22 12:16:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=3.9085 [2026-10-22 12:16:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6202 [2026-10-22 12:17:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0211 [2026-10-22 12:17:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1372 [2026-10-22 12:17:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3164 [2026-10-22 12:17:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1241 [2026-10-22 12:17:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.1223 [2026-10-22 12:17:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7813 [2026-10-22 12:17:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3082 [2026-10-22 12:17:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3292 [2026-10-22 12:17:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1690 [2026-10-22 12:17:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9745 [2026-10-22 12:17:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4472 [2026-10-22 12:17:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1731 [2026-10-22 12:17:53][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,629 | MFU: 1.8% [2026-10-22 12:17:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0617 [2026-10-22 12:17:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7297 [2026-10-22 12:17:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9463 [2026-10-22 12:17:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3607 [2026-10-22 12:17:59][INFO][training.pretrain_engine] [Ph2|Step 740] loss=4.7046 | ppl=110.45 | aux=0.02505 | lr=2.96e-04 | gnorm=0.875 | tok/s=29,343 | MFU=7.0% | step=142942ms | tokens=1.578B | mem=4.2/32.7GB | cost=$60.72/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 19:21 (127.1h) | Phase3: SKIPPED | SFT: Oct 28 00:05 (131.8h) | DPO (Final): Oct 28 00:39 (132.4h) [2026-10-22 12:17:59][INFO][training.pretrain_engine] Starting global step 741 (accumulating 32 micro-batches)... [2026-10-22 12:17:59][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 12:18:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2916 [2026-10-22 12:18:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5449 [2026-10-22 12:18:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9577 [2026-10-22 12:18:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9946 [2026-10-22 12:18:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5606 [2026-10-22 12:18:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8272 [2026-10-22 12:18:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2927 [2026-10-22 12:18:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4226 [2026-10-22 12:18:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0770 [2026-10-22 12:18:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0441 [2026-10-22 12:18:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0962 [2026-10-22 12:18:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9631 [2026-10-22 12:19:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0799 [2026-10-22 12:19:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6096 [2026-10-22 12:19:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6740 [2026-10-22 12:19:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4407 [2026-10-22 12:19:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2758 [2026-10-22 12:19:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0071 [2026-10-22 12:19:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9199 [2026-10-22 12:19:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2542 [2026-10-22 12:19:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5898 [2026-10-22 12:19:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9283 [2026-10-22 12:19:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8068 [2026-10-22 12:19:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6439 [2026-10-22 12:20:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1338 [2026-10-22 12:20:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8452 [2026-10-22 12:20:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1349 [2026-10-22 12:20:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8750 [2026-10-22 12:20:16][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,617 | MFU: 1.8% [2026-10-22 12:20:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2849 [2026-10-22 12:20:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6559 [2026-10-22 12:20:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7760 [2026-10-22 12:20:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2633 [2026-10-22 12:20:22][INFO][training.pretrain_engine] [Ph2|Step 741] loss=4.7468 | ppl=115.21 | aux=0.02505 | lr=2.96e-04 | gnorm=1.242 | tok/s=29,343 | MFU=7.0% | step=142939ms | tokens=1.579B | mem=4.2/32.7GB | cost=$60.80/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 19:23 (127.0h) | Phase3: SKIPPED | SFT: Oct 28 00:07 (131.8h) | DPO (Final): Oct 28 00:41 (132.3h) [2026-10-22 12:20:22][INFO][training.pretrain_engine] Starting global step 742 (accumulating 32 micro-batches)... [2026-10-22 12:20:22][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 12:20:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4313 [2026-10-22 12:20:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4685 [2026-10-22 12:20:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7121 [2026-10-22 12:20:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9496 [2026-10-22 12:20:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.1226 [2026-10-22 12:20:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.7129 [2026-10-22 12:20:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1775 [2026-10-22 12:20:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.0480 [2026-10-22 12:21:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2910 [2026-10-22 12:21:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=3.8240 [2026-10-22 12:21:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5341 [2026-10-22 12:21:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3916 [2026-10-22 12:21:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1068 [2026-10-22 12:21:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1608 [2026-10-22 12:21:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2687 [2026-10-22 12:21:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0571 [2026-10-22 12:21:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5364 [2026-10-22 12:21:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9113 [2026-10-22 12:21:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5007 [2026-10-22 12:21:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7243 [2026-10-22 12:22:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8658 [2026-10-22 12:22:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6192 [2026-10-22 12:22:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9008 [2026-10-22 12:22:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6451 [2026-10-22 12:22:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3271 [2026-10-22 12:22:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3047 [2026-10-22 12:22:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.8699 [2026-10-22 12:22:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9226 [2026-10-22 12:22:39][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,609 | MFU: 1.8% [2026-10-22 12:22:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0082 [2026-10-22 12:22:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9674 [2026-10-22 12:22:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7225 [2026-10-22 12:22:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8467 [2026-10-22 12:22:44][INFO][training.pretrain_engine] [Ph2|Step 742] loss=4.7793 | ppl=119.02 | aux=0.02506 | lr=2.96e-04 | gnorm=0.910 | tok/s=29,346 | MFU=7.0% | step=142928ms | tokens=1.580B | mem=4.2/32.7GB | cost=$60.88/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 19:24 (127.0h) | Phase3: SKIPPED | SFT: Oct 28 00:08 (131.8h) | DPO (Final): Oct 28 00:42 (132.3h) [2026-10-22 12:22:44][INFO][training.pretrain_engine] Starting global step 743 (accumulating 32 micro-batches)... [2026-10-22 12:22:44][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 12:23:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.9039 [2026-10-22 12:23:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4474 [2026-10-22 12:23:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8419 [2026-10-22 12:23:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7087 [2026-10-22 12:23:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.6098 [2026-10-22 12:23:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2117 [2026-10-22 12:23:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7215 [2026-10-22 12:23:20][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3769 [2026-10-22 12:23:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.6386 [2026-10-22 12:23:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4020 [2026-10-22 12:23:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7784 [2026-10-22 12:23:38][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7821 [2026-10-22 12:23:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0932 [2026-10-22 12:23:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3200 [2026-10-22 12:23:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3595 [2026-10-22 12:23:56][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4225 [2026-10-22 12:24:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4383 [2026-10-22 12:24:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6182 [2026-10-22 12:24:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5130 [2026-10-22 12:24:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.2475 [2026-10-22 12:24:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3169 [2026-10-22 12:24:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0621 [2026-10-22 12:24:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9715 [2026-10-22 12:24:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1770 [2026-10-22 12:24:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1706 [2026-10-22 12:24:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2083 [2026-10-22 12:24:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1106 [2026-10-22 12:24:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9248 [2026-10-22 12:25:02][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,612 | MFU: 1.8% [2026-10-22 12:25:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8999 [2026-10-22 12:25:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5785 [2026-10-22 12:25:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7339 [2026-10-22 12:25:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8988 [2026-10-22 12:25:07][INFO][training.pretrain_engine] [Ph2|Step 743] loss=4.7126 | ppl=111.35 | aux=0.02506 | lr=2.96e-04 | gnorm=0.977 | tok/s=29,347 | MFU=7.0% | step=142923ms | tokens=1.581B | mem=4.2/32.7GB | cost=$60.96/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 19:25 (127.0h) | Phase3: SKIPPED | SFT: Oct 28 00:09 (131.7h) | DPO (Final): Oct 28 00:44 (132.3h) [2026-10-22 12:25:07][INFO][training.pretrain_engine] Starting global step 744 (accumulating 32 micro-batches)... [2026-10-22 12:25:07][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 12:25:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2285 [2026-10-22 12:25:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0796 [2026-10-22 12:25:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1696 [2026-10-22 12:25:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3583 [2026-10-22 12:25:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3178 [2026-10-22 12:25:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2644 [2026-10-22 12:25:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.5080 [2026-10-22 12:25:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2130 [2026-10-22 12:26:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2642 [2026-10-22 12:26:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4047 [2026-10-22 12:26:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8691 [2026-10-22 12:26:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7566 [2026-10-22 12:26:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5376 [2026-10-22 12:26:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9690 [2026-10-22 12:26:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5553 [2026-10-22 12:26:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7207 [2026-10-22 12:26:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=3.9464 [2026-10-22 12:26:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.0718 [2026-10-22 12:26:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5347 [2026-10-22 12:26:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.1312 [2026-10-22 12:26:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2679 [2026-10-22 12:26:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6026 [2026-10-22 12:26:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.1502 [2026-10-22 12:26:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5379 [2026-10-22 12:27:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3559 [2026-10-22 12:27:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4098 [2026-10-22 12:27:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1631 [2026-10-22 12:27:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4989 [2026-10-22 12:27:24][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,626 | MFU: 1.8% [2026-10-22 12:27:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7152 [2026-10-22 12:27:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8543 [2026-10-22 12:27:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9507 [2026-10-22 12:27:30][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7076 [2026-10-22 12:27:30][INFO][training.pretrain_engine] [Ph2|Step 744] loss=4.7475 | ppl=115.30 | aux=0.02506 | lr=2.96e-04 | gnorm=0.914 | tok/s=29,348 | MFU=7.0% | step=142918ms | tokens=1.582B | mem=4.2/32.7GB | cost=$61.04/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 19:27 (127.0h) | Phase3: SKIPPED | SFT: Oct 28 00:11 (131.7h) | DPO (Final): Oct 28 00:45 (132.3h) [2026-10-22 12:27:30][INFO][training.pretrain_engine] Starting global step 745 (accumulating 32 micro-batches)... [2026-10-22 12:27:30][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 12:27:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1077 [2026-10-22 12:27:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0733 [2026-10-22 12:27:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0518 [2026-10-22 12:27:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4699 [2026-10-22 12:28:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0300 [2026-10-22 12:28:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5552 [2026-10-22 12:28:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2862 [2026-10-22 12:28:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0374 [2026-10-22 12:28:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1669 [2026-10-22 12:28:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1441 [2026-10-22 12:28:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1286 [2026-10-22 12:28:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7382 [2026-10-22 12:28:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9742 [2026-10-22 12:28:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5766 [2026-10-22 12:28:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2907 [2026-10-22 12:28:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5120 [2026-10-22 12:28:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1462 [2026-10-22 12:28:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3745 [2026-10-22 12:28:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9099 [2026-10-22 12:28:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0433 [2026-10-22 12:29:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9012 [2026-10-22 12:29:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0207 [2026-10-22 12:29:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7750 [2026-10-22 12:29:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6543 [2026-10-22 12:29:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5626 [2026-10-22 12:29:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.1518 [2026-10-22 12:29:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2927 [2026-10-22 12:29:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.0253 [2026-10-22 12:29:47][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,625 | MFU: 1.8% [2026-10-22 12:29:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7006 [2026-10-22 12:29:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3492 [2026-10-22 12:29:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7447 [2026-10-22 12:29:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7707 [2026-10-22 12:29:52][INFO][training.pretrain_engine] [Ph2|Step 745] loss=4.8260 | ppl=124.71 | aux=0.02506 | lr=2.96e-04 | gnorm=0.719 | tok/s=29,348 | MFU=7.0% | step=142916ms | tokens=1.583B | mem=4.2/32.7GB | cost=$61.13/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 19:29 (127.0h) | Phase3: SKIPPED | SFT: Oct 28 00:13 (131.7h) | DPO (Final): Oct 28 00:47 (132.3h) [2026-10-22 12:29:52][INFO][training.pretrain_engine] Starting global step 746 (accumulating 32 micro-batches)... [2026-10-22 12:29:52][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 12:30:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5728 [2026-10-22 12:30:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6082 [2026-10-22 12:30:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5846 [2026-10-22 12:30:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5782 [2026-10-22 12:30:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2114 [2026-10-22 12:30:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3826 [2026-10-22 12:30:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8550 [2026-10-22 12:30:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8733 [2026-10-22 12:30:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7796 [2026-10-22 12:30:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8999 [2026-10-22 12:30:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4006 [2026-10-22 12:30:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0698 [2026-10-22 12:31:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4106 [2026-10-22 12:31:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7212 [2026-10-22 12:31:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7428 [2026-10-22 12:31:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9710 [2026-10-22 12:31:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3152 [2026-10-22 12:31:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9071 [2026-10-22 12:31:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5928 [2026-10-22 12:31:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5347 [2026-10-22 12:31:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4017 [2026-10-22 12:31:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.1428 [2026-10-22 12:31:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3116 [2026-10-22 12:31:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.0721 [2026-10-22 12:31:56][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9753 [2026-10-22 12:31:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0659 [2026-10-22 12:31:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2533 [2026-10-22 12:31:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9084 [2026-10-22 12:32:10][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,620 | MFU: 1.8% [2026-10-22 12:32:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1102 [2026-10-22 12:32:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3377 [2026-10-22 12:32:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4076 [2026-10-22 12:32:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7090 [2026-10-22 12:32:15][INFO][training.pretrain_engine] [Ph2|Step 746] loss=4.5625 | ppl=95.82 | aux=0.02506 | lr=2.96e-04 | gnorm=0.703 | tok/s=29,348 | MFU=7.0% | step=142918ms | tokens=1.584B | mem=4.2/32.7GB | cost=$61.21/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 19:31 (127.0h) | Phase3: SKIPPED | SFT: Oct 28 00:15 (131.7h) | DPO (Final): Oct 28 00:49 (132.3h) [2026-10-22 12:32:15][INFO][training.pretrain_engine] Starting global step 747 (accumulating 32 micro-batches)... [2026-10-22 12:32:15][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 12:32:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4896 [2026-10-22 12:32:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5537 [2026-10-22 12:32:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8096 [2026-10-22 12:32:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7926 [2026-10-22 12:32:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5963 [2026-10-22 12:32:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3749 [2026-10-22 12:32:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4036 [2026-10-22 12:32:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6626 [2026-10-22 12:33:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2240 [2026-10-22 12:33:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8631 [2026-10-22 12:33:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7308 [2026-10-22 12:33:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3356 [2026-10-22 12:33:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8513 [2026-10-22 12:33:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0745 [2026-10-22 12:33:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4391 [2026-10-22 12:33:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0329 [2026-10-22 12:33:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.0683 [2026-10-22 12:33:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.1389 [2026-10-22 12:33:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.1712 [2026-10-22 12:33:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5540 [2026-10-22 12:34:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7098 [2026-10-22 12:34:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9385 [2026-10-22 12:34:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7197 [2026-10-22 12:34:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5751 [2026-10-22 12:34:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0936 [2026-10-22 12:34:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8114 [2026-10-22 12:34:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6155 [2026-10-22 12:34:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8089 [2026-10-22 12:34:33][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,619 | MFU: 1.8% [2026-10-22 12:34:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5226 [2026-10-22 12:34:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4151 [2026-10-22 12:34:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4417 [2026-10-22 12:34:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=3.9660 [2026-10-22 12:34:38][INFO][training.pretrain_engine] [Ph2|Step 747] loss=4.6608 | ppl=105.72 | aux=0.02505 | lr=2.96e-04 | gnorm=0.715 | tok/s=29,347 | MFU=7.0% | step=142919ms | tokens=1.585B | mem=4.2/32.7GB | cost=$61.29/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 19:32 (127.0h) | Phase3: SKIPPED | SFT: Oct 28 00:16 (131.7h) | DPO (Final): Oct 28 00:50 (132.3h) [2026-10-22 12:34:41][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step747 | Tokens: 1.585B | Spent: $61.29 | Remaining: $938.71 [2026-10-22 12:34:41][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step747 [2026-10-22 12:34:41][INFO][training.pretrain_engine] Starting global step 748 (accumulating 32 micro-batches)... [2026-10-22 12:34:41][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 12:34:41][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-22 12:34:41][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-22 12:34:41][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-22 12:34:41][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step747 to Phase-Technologies/forge-500m-pretrain (path: phase2_step747)... [2026-10-22 12:34:42][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-22 12:34:51][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-22 12:34:51][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step747 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-22 12:34:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3706 [2026-10-22 12:34:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5896 [2026-10-22 12:34:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.8300 [2026-10-22 12:34:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.8523 [2026-10-22 12:35:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4045 [2026-10-22 12:35:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5361 [2026-10-22 12:35:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8013 [2026-10-22 12:35:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6817 [2026-10-22 12:35:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8404 [2026-10-22 12:35:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0630 [2026-10-22 12:35:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1452 [2026-10-22 12:35:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4287 [2026-10-22 12:35:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6732 [2026-10-22 12:35:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4674 [2026-10-22 12:35:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7148 [2026-10-22 12:35:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=3.8287 [2026-10-22 12:36:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.1158 [2026-10-22 12:36:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.0637 [2026-10-22 12:36:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.0100 [2026-10-22 12:36:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=3.8553 [2026-10-22 12:36:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4098 [2026-10-22 12:36:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8320 [2026-10-22 12:36:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6030 [2026-10-22 12:36:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8638 [2026-10-22 12:36:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4402 [2026-10-22 12:36:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4789 [2026-10-22 12:36:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5270 [2026-10-22 12:36:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3715 [2026-10-22 12:36:59][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,608 | MFU: 1.8% [2026-10-22 12:37:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3582 [2026-10-22 12:37:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9902 [2026-10-22 12:37:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5587 [2026-10-22 12:37:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9059 [2026-10-22 12:37:03][INFO][training.pretrain_engine] [Ph2|Step 748] loss=4.6948 | ppl=109.38 | aux=0.02506 | lr=2.96e-04 | gnorm=0.562 | tok/s=29,349 | MFU=7.0% | step=142909ms | tokens=1.586B | mem=4.2/32.7GB | cost=$61.37/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 19:34 (127.0h) | Phase3: SKIPPED | SFT: Oct 28 00:18 (131.7h) | DPO (Final): Oct 28 00:52 (132.3h) [2026-10-22 12:37:03][INFO][training.pretrain_engine] Starting global step 749 (accumulating 32 micro-batches)... [2026-10-22 12:37:03][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 12:37:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9076 [2026-10-22 12:37:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8545 [2026-10-22 12:37:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4554 [2026-10-22 12:37:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1919 [2026-10-22 12:37:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0079 [2026-10-22 12:37:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7812 [2026-10-22 12:37:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5926 [2026-10-22 12:37:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8274 [2026-10-22 12:37:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8620 [2026-10-22 12:37:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5584 [2026-10-22 12:37:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5965 [2026-10-22 12:37:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7808 [2026-10-22 12:38:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8794 [2026-10-22 12:38:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6927 [2026-10-22 12:38:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6683 [2026-10-22 12:38:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8500 [2026-10-22 12:38:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2631 [2026-10-22 12:38:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8086 [2026-10-22 12:38:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2312 [2026-10-22 12:38:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3347 [2026-10-22 12:38:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=3.9622 [2026-10-22 12:38:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4404 [2026-10-22 12:38:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6061 [2026-10-22 12:38:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4520 [2026-10-22 12:39:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6367 [2026-10-22 12:39:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3731 [2026-10-22 12:39:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5844 [2026-10-22 12:39:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8686 [2026-10-22 12:39:21][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,618 | MFU: 1.8% [2026-10-22 12:39:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8524 [2026-10-22 12:39:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0480 [2026-10-22 12:39:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9022 [2026-10-22 12:39:26][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9689 [2026-10-22 12:39:26][INFO][training.pretrain_engine] [Ph2|Step 749] loss=4.7528 | ppl=115.91 | aux=0.02506 | lr=2.96e-04 | gnorm=0.633 | tok/s=29,350 | MFU=7.0% | step=142909ms | tokens=1.588B | mem=4.2/32.7GB | cost=$61.45/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 19:35 (126.9h) | Phase3: SKIPPED | SFT: Oct 28 00:19 (131.7h) | DPO (Final): Oct 28 00:53 (132.2h) [2026-10-22 12:39:26][INFO][training.pretrain_engine] Starting global step 750 (accumulating 32 micro-batches)... [2026-10-22 12:39:26][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 12:39:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.0018 [2026-10-22 12:39:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.0626 [2026-10-22 12:39:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.0485 [2026-10-22 12:39:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.1169 [2026-10-22 12:40:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9675 [2026-10-22 12:40:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5469 [2026-10-22 12:40:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7347 [2026-10-22 12:40:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3943 [2026-10-22 12:40:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4057 [2026-10-22 12:40:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7083 [2026-10-22 12:40:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5702 [2026-10-22 12:40:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8397 [2026-10-22 12:40:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2726 [2026-10-22 12:40:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=3.7014 [2026-10-22 12:40:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6750 [2026-10-22 12:40:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2559 [2026-10-22 12:40:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3644 [2026-10-22 12:40:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7012 [2026-10-22 12:40:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1396 [2026-10-22 12:40:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6480 [2026-10-22 12:41:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9888 [2026-10-22 12:41:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8005 [2026-10-22 12:41:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0375 [2026-10-22 12:41:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6442 [2026-10-22 12:41:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=3.8661 [2026-10-22 12:41:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3037 [2026-10-22 12:41:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2176 [2026-10-22 12:41:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.0001 [2026-10-22 12:41:44][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,607 | MFU: 1.8% [2026-10-22 12:41:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.0964 [2026-10-22 12:41:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6626 [2026-10-22 12:41:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7540 [2026-10-22 12:41:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7010 [2026-10-22 12:41:49][INFO][training.pretrain_engine] [Ph2|Step 750] loss=4.6793 | ppl=107.69 | aux=0.02505 | lr=2.96e-04 | gnorm=0.594 | tok/s=29,350 | MFU=7.0% | step=142907ms | tokens=1.589B | mem=4.2/32.7GB | cost=$61.54/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 19:37 (126.9h) | Phase3: SKIPPED | SFT: Oct 28 00:21 (131.7h) | DPO (Final): Oct 28 00:55 (132.2h) [2026-10-22 12:41:49][INFO][training.pretrain_engine] Starting global step 751 (accumulating 32 micro-batches)... [2026-10-22 12:41:49][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 12:42:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8849 [2026-10-22 12:42:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7337 [2026-10-22 12:42:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9178 [2026-10-22 12:42:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8859 [2026-10-22 12:42:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4634 [2026-10-22 12:42:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7493 [2026-10-22 12:42:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7603 [2026-10-22 12:42:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5096 [2026-10-22 12:42:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9653 [2026-10-22 12:42:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8248 [2026-10-22 12:42:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8231 [2026-10-22 12:42:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6279 [2026-10-22 12:43:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2833 [2026-10-22 12:43:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6860 [2026-10-22 12:43:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6826 [2026-10-22 12:43:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.0990 [2026-10-22 12:43:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7981 [2026-10-22 12:43:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.1629 [2026-10-22 12:43:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4954 [2026-10-22 12:43:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8865 [2026-10-22 12:43:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.0100 [2026-10-22 12:43:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4958 [2026-10-22 12:43:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5743 [2026-10-22 12:43:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3921 [2026-10-22 12:43:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3500 [2026-10-22 12:43:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9841 [2026-10-22 12:43:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2497 [2026-10-22 12:43:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8471 [2026-10-22 12:44:07][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,601 | MFU: 1.8% [2026-10-22 12:44:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6206 [2026-10-22 12:44:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9064 [2026-10-22 12:44:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0156 [2026-10-22 12:44:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8167 [2026-10-22 12:44:12][INFO][training.pretrain_engine] [Ph2|Step 751] loss=4.7055 | ppl=110.55 | aux=0.02505 | lr=2.96e-04 | gnorm=0.523 | tok/s=29,351 | MFU=7.0% | step=142903ms | tokens=1.590B | mem=4.2/32.7GB | cost=$61.62/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 19:39 (126.9h) | Phase3: SKIPPED | SFT: Oct 28 00:23 (131.6h) | DPO (Final): Oct 28 00:57 (132.2h) [2026-10-22 12:44:12][INFO][training.pretrain_engine] Starting global step 752 (accumulating 32 micro-batches)... [2026-10-22 12:44:12][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 12:44:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0077 [2026-10-22 12:44:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1161 [2026-10-22 12:44:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1234 [2026-10-22 12:44:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3312 [2026-10-22 12:44:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1786 [2026-10-22 12:44:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8517 [2026-10-22 12:44:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8486 [2026-10-22 12:44:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6880 [2026-10-22 12:45:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9720 [2026-10-22 12:45:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8822 [2026-10-22 12:45:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6562 [2026-10-22 12:45:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2743 [2026-10-22 12:45:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2291 [2026-10-22 12:45:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1205 [2026-10-22 12:45:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4893 [2026-10-22 12:45:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1356 [2026-10-22 12:45:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8098 [2026-10-22 12:45:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9156 [2026-10-22 12:45:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6740 [2026-10-22 12:45:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8930 [2026-10-22 12:45:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7801 [2026-10-22 12:45:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6485 [2026-10-22 12:45:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2801 [2026-10-22 12:45:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6256 [2026-10-22 12:46:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9278 [2026-10-22 12:46:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9370 [2026-10-22 12:46:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8461 [2026-10-22 12:46:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9978 [2026-10-22 12:46:30][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,601 | MFU: 1.8% [2026-10-22 12:46:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5123 [2026-10-22 12:46:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5424 [2026-10-22 12:46:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6471 [2026-10-22 12:46:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9463 [2026-10-22 12:46:35][INFO][training.pretrain_engine] [Ph2|Step 752] loss=4.6996 | ppl=109.91 | aux=0.02506 | lr=2.96e-04 | gnorm=0.547 | tok/s=29,352 | MFU=7.0% | step=142898ms | tokens=1.591B | mem=4.2/32.7GB | cost=$61.70/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 19:40 (126.9h) | Phase3: SKIPPED | SFT: Oct 28 00:24 (131.6h) | DPO (Final): Oct 28 00:58 (132.2h) [2026-10-22 12:46:35][INFO][training.pretrain_engine] Starting global step 753 (accumulating 32 micro-batches)... [2026-10-22 12:46:35][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 12:46:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5633 [2026-10-22 12:46:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.1643 [2026-10-22 12:46:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3925 [2026-10-22 12:46:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7487 [2026-10-22 12:47:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9639 [2026-10-22 12:47:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8184 [2026-10-22 12:47:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9197 [2026-10-22 12:47:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8201 [2026-10-22 12:47:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4780 [2026-10-22 12:47:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7584 [2026-10-22 12:47:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7425 [2026-10-22 12:47:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4710 [2026-10-22 12:47:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7671 [2026-10-22 12:47:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8725 [2026-10-22 12:47:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0997 [2026-10-22 12:47:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3956 [2026-10-22 12:48:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3955 [2026-10-22 12:48:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5632 [2026-10-22 12:48:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4135 [2026-10-22 12:48:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4211 [2026-10-22 12:48:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5617 [2026-10-22 12:48:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5008 [2026-10-22 12:48:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0201 [2026-10-22 12:48:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6485 [2026-10-22 12:48:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9479 [2026-10-22 12:48:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3121 [2026-10-22 12:48:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8774 [2026-10-22 12:48:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1427 [2026-10-22 12:48:53][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,602 | MFU: 1.8% [2026-10-22 12:48:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=3.5495 [2026-10-22 12:48:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=3.2494 [2026-10-22 12:48:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7464 [2026-10-22 12:48:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.0609 [2026-10-22 12:48:58][INFO][training.pretrain_engine] [Ph2|Step 753] loss=4.6326 | ppl=102.78 | aux=0.02506 | lr=2.96e-04 | gnorm=0.523 | tok/s=29,353 | MFU=7.0% | step=142892ms | tokens=1.592B | mem=4.2/32.7GB | cost=$61.78/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 19:42 (126.9h) | Phase3: SKIPPED | SFT: Oct 28 00:26 (131.6h) | DPO (Final): Oct 28 01:00 (132.2h) [2026-10-22 12:48:58][INFO][training.pretrain_engine] Starting global step 754 (accumulating 32 micro-batches)... [2026-10-22 12:48:58][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 12:49:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6475 [2026-10-22 12:49:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3976 [2026-10-22 12:49:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5953 [2026-10-22 12:49:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3939 [2026-10-22 12:49:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4439 [2026-10-22 12:49:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2844 [2026-10-22 12:49:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4015 [2026-10-22 12:49:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1023 [2026-10-22 12:49:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9688 [2026-10-22 12:49:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6112 [2026-10-22 12:49:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9336 [2026-10-22 12:49:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4792 [2026-10-22 12:50:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6317 [2026-10-22 12:50:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0442 [2026-10-22 12:50:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6004 [2026-10-22 12:50:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5560 [2026-10-22 12:50:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4998 [2026-10-22 12:50:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8300 [2026-10-22 12:50:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9775 [2026-10-22 12:50:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6096 [2026-10-22 12:50:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7844 [2026-10-22 12:50:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8790 [2026-10-22 12:50:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2731 [2026-10-22 12:50:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7529 [2026-10-22 12:51:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2495 [2026-10-22 12:51:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0887 [2026-10-22 12:51:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9735 [2026-10-22 12:51:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6371 [2026-10-22 12:51:15][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,621 | MFU: 1.8% [2026-10-22 12:51:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0302 [2026-10-22 12:51:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=3.6668 [2026-10-22 12:51:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1644 [2026-10-22 12:51:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6026 [2026-10-22 12:51:20][INFO][training.pretrain_engine] [Ph2|Step 754] loss=4.7287 | ppl=113.15 | aux=0.02506 | lr=2.96e-04 | gnorm=0.512 | tok/s=29,355 | MFU=7.0% | step=142881ms | tokens=1.593B | mem=4.2/32.7GB | cost=$61.86/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 19:43 (126.9h) | Phase3: SKIPPED | SFT: Oct 28 00:27 (131.6h) | DPO (Final): Oct 28 01:01 (132.2h) [2026-10-22 12:51:20][INFO][training.pretrain_engine] Starting global step 755 (accumulating 32 micro-batches)... [2026-10-22 12:51:20][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 12:51:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2800 [2026-10-22 12:51:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0784 [2026-10-22 12:51:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1794 [2026-10-22 12:51:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7152 [2026-10-22 12:51:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3117 [2026-10-22 12:51:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.8472 [2026-10-22 12:51:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.6688 [2026-10-22 12:51:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2396 [2026-10-22 12:52:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.1432 [2026-10-22 12:52:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8627 [2026-10-22 12:52:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2340 [2026-10-22 12:52:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8099 [2026-10-22 12:52:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.1753 [2026-10-22 12:52:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8066 [2026-10-22 12:52:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5241 [2026-10-22 12:52:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5310 [2026-10-22 12:52:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4694 [2026-10-22 12:52:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5300 [2026-10-22 12:52:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8096 [2026-10-22 12:52:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6827 [2026-10-22 12:53:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5550 [2026-10-22 12:53:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7673 [2026-10-22 12:53:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9493 [2026-10-22 12:53:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4465 [2026-10-22 12:53:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9876 [2026-10-22 12:53:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8566 [2026-10-22 12:53:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9524 [2026-10-22 12:53:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2004 [2026-10-22 12:53:38][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,603 | MFU: 1.8% [2026-10-22 12:53:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9469 [2026-10-22 12:53:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4141 [2026-10-22 12:53:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1158 [2026-10-22 12:53:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3521 [2026-10-22 12:53:43][INFO][training.pretrain_engine] [Ph2|Step 755] loss=4.7749 | ppl=118.50 | aux=0.02505 | lr=2.96e-04 | gnorm=0.516 | tok/s=29,358 | MFU=7.0% | step=142869ms | tokens=1.594B | mem=4.2/32.7GB | cost=$61.94/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 19:44 (126.8h) | Phase3: SKIPPED | SFT: Oct 28 00:28 (131.6h) | DPO (Final): Oct 28 01:02 (132.1h) [2026-10-22 12:53:43][INFO][training.pretrain_engine] Starting global step 756 (accumulating 32 micro-batches)... [2026-10-22 12:53:43][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 12:54:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3016 [2026-10-22 12:54:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3381 [2026-10-22 12:54:00][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5270 [2026-10-22 12:54:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2165 [2026-10-22 12:54:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7228 [2026-10-22 12:54:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8326 [2026-10-22 12:54:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3939 [2026-10-22 12:54:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5781 [2026-10-22 12:54:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8031 [2026-10-22 12:54:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5287 [2026-10-22 12:54:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0184 [2026-10-22 12:54:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7080 [2026-10-22 12:54:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7003 [2026-10-22 12:54:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1869 [2026-10-22 12:54:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7164 [2026-10-22 12:54:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7410 [2026-10-22 12:55:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7363 [2026-10-22 12:55:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.0600 [2026-10-22 12:55:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4584 [2026-10-22 12:55:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6531 [2026-10-22 12:55:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5411 [2026-10-22 12:55:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3528 [2026-10-22 12:55:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0016 [2026-10-22 12:55:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8555 [2026-10-22 12:55:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2781 [2026-10-22 12:55:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3469 [2026-10-22 12:55:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7370 [2026-10-22 12:55:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5321 [2026-10-22 12:56:00][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,629 | MFU: 1.8% [2026-10-22 12:56:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.0807 [2026-10-22 12:56:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7142 [2026-10-22 12:56:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6853 [2026-10-22 12:56:05][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9008 [2026-10-22 12:56:05][INFO][training.pretrain_engine] [Ph2|Step 756] loss=4.6469 | ppl=104.26 | aux=0.02506 | lr=2.96e-04 | gnorm=0.543 | tok/s=29,358 | MFU=7.0% | step=142866ms | tokens=1.595B | mem=4.2/32.7GB | cost=$62.03/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 19:46 (126.8h) | Phase3: SKIPPED | SFT: Oct 28 00:30 (131.6h) | DPO (Final): Oct 28 01:04 (132.1h) [2026-10-22 12:56:08][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step756 | Tokens: 1.595B | Spent: $62.03 | Remaining: $937.97 [2026-10-22 12:56:08][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step756 [2026-10-22 12:56:08][INFO][training.pretrain_engine] Starting global step 757 (accumulating 32 micro-batches)... [2026-10-22 12:56:08][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 12:56:08][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-22 12:56:08][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-22 12:56:08][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-22 12:56:08][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step756 to Phase-Technologies/forge-500m-pretrain (path: phase2_step756)... [2026-10-22 12:56:09][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-22 12:56:18][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-22 12:56:18][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step756 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-22 12:56:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2475 [2026-10-22 12:56:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1261 [2026-10-22 12:56:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9621 [2026-10-22 12:56:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0813 [2026-10-22 12:56:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6933 [2026-10-22 12:56:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7232 [2026-10-22 12:56:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6368 [2026-10-22 12:56:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7068 [2026-10-22 12:56:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8085 [2026-10-22 12:56:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3833 [2026-10-22 12:56:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1582 [2026-10-22 12:57:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0841 [2026-10-22 12:57:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4541 [2026-10-22 12:57:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5816 [2026-10-22 12:57:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7185 [2026-10-22 12:57:19][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9540 [2026-10-22 12:57:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3737 [2026-10-22 12:57:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6269 [2026-10-22 12:57:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8431 [2026-10-22 12:57:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4205 [2026-10-22 12:57:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8733 [2026-10-22 12:57:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9199 [2026-10-22 12:57:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8551 [2026-10-22 12:57:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9246 [2026-10-22 12:58:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0031 [2026-10-22 12:58:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6741 [2026-10-22 12:58:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6740 [2026-10-22 12:58:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2776 [2026-10-22 12:58:27][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,559 | MFU: 1.8% [2026-10-22 12:58:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4082 [2026-10-22 12:58:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3065 [2026-10-22 12:58:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7977 [2026-10-22 12:58:31][INFO][training.pretrain_engine] [Ph2|Step 757] loss=4.7312 | ppl=113.44 | aux=0.02506 | lr=2.96e-04 | gnorm=0.479 | tok/s=29,357 | MFU=7.0% | step=142871ms | tokens=1.596B | mem=4.2/32.7GB | cost=$62.11/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 19:48 (126.8h) | Phase3: SKIPPED | SFT: Oct 28 00:32 (131.6h) | DPO (Final): Oct 28 01:06 (132.1h) [2026-10-22 12:58:31][INFO][training.pretrain_engine] Starting global step 758 (accumulating 32 micro-batches)... [2026-10-22 12:58:31][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 12:58:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5176 [2026-10-22 12:58:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2258 [2026-10-22 12:58:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2040 [2026-10-22 12:58:50][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7670 [2026-10-22 12:58:51][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9145 [2026-10-22 12:59:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1104 [2026-10-22 12:59:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8852 [2026-10-22 12:59:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6421 [2026-10-22 12:59:08][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8374 [2026-10-22 12:59:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9056 [2026-10-22 12:59:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6825 [2026-10-22 12:59:26][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8590 [2026-10-22 12:59:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0969 [2026-10-22 12:59:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7480 [2026-10-22 12:59:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0124 [2026-10-22 12:59:44][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4090 [2026-10-22 12:59:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7439 [2026-10-22 13:00:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8707 [2026-10-22 13:00:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8027 [2026-10-22 13:00:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0007 [2026-10-22 13:00:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7431 [2026-10-22 13:00:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6013 [2026-10-22 13:00:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5449 [2026-10-22 13:00:19][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2652 [2026-10-22 13:00:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8591 [2026-10-22 13:00:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1223 [2026-10-22 13:00:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0265 [2026-10-22 13:00:37][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1252 [2026-10-22 13:00:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1362 [2026-10-22 13:00:52][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,448 | MFU: 1.8% [2026-10-22 13:00:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0567 [2026-10-22 13:00:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2735 [2026-10-22 13:00:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7691 [2026-10-22 13:00:57][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7849 [2026-10-22 13:00:57][INFO][training.pretrain_engine] [Ph2|Step 758] loss=4.7433 | ppl=114.82 | aux=0.02505 | lr=2.96e-04 | gnorm=0.602 | tok/s=29,351 | MFU=7.0% | step=142902ms | tokens=1.597B | mem=4.2/32.7GB | cost=$62.19/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 19:51 (126.8h) | Phase3: SKIPPED | SFT: Oct 28 00:35 (131.6h) | DPO (Final): Oct 28 01:09 (132.1h) [2026-10-22 13:00:57][INFO][training.pretrain_engine] Starting global step 759 (accumulating 32 micro-batches)... [2026-10-22 13:00:57][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 13:01:14][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7789 [2026-10-22 13:01:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1046 [2026-10-22 13:01:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9667 [2026-10-22 13:01:15][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0673 [2026-10-22 13:01:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4580 [2026-10-22 13:01:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2386 [2026-10-22 13:01:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6239 [2026-10-22 13:01:32][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5165 [2026-10-22 13:01:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=3.8151 [2026-10-22 13:01:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2268 [2026-10-22 13:01:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2406 [2026-10-22 13:01:50][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4748 [2026-10-22 13:02:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8944 [2026-10-22 13:02:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9946 [2026-10-22 13:02:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6704 [2026-10-22 13:02:08][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9930 [2026-10-22 13:02:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0246 [2026-10-22 13:02:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2767 [2026-10-22 13:02:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1897 [2026-10-22 13:02:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0074 [2026-10-22 13:02:43][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8593 [2026-10-22 13:02:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9125 [2026-10-22 13:02:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0704 [2026-10-22 13:02:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0811 [2026-10-22 13:03:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8252 [2026-10-22 13:03:01][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0553 [2026-10-22 13:03:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0635 [2026-10-22 13:03:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8917 [2026-10-22 13:03:15][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,608 | MFU: 1.8% [2026-10-22 13:03:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3056 [2026-10-22 13:03:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9943 [2026-10-22 13:03:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9243 [2026-10-22 13:03:20][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4474 [2026-10-22 13:03:20][INFO][training.pretrain_engine] [Ph2|Step 759] loss=4.7101 | ppl=111.06 | aux=0.02506 | lr=2.96e-04 | gnorm=0.539 | tok/s=29,351 | MFU=7.0% | step=142901ms | tokens=1.598B | mem=4.2/32.7GB | cost=$62.27/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 19:53 (126.8h) | Phase3: SKIPPED | SFT: Oct 28 00:37 (131.6h) | DPO (Final): Oct 28 01:11 (132.1h) [2026-10-22 13:03:20][INFO][training.pretrain_engine] Starting global step 760 (accumulating 32 micro-batches)... [2026-10-22 13:03:20][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 13:03:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5192 [2026-10-22 13:03:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6621 [2026-10-22 13:03:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2019 [2026-10-22 13:03:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4938 [2026-10-22 13:03:55][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7103 [2026-10-22 13:03:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6377 [2026-10-22 13:03:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7756 [2026-10-22 13:03:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6535 [2026-10-22 13:04:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0741 [2026-10-22 13:04:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7290 [2026-10-22 13:04:13][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0115 [2026-10-22 13:04:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0465 [2026-10-22 13:04:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9010 [2026-10-22 13:04:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4625 [2026-10-22 13:04:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4255 [2026-10-22 13:04:31][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.0023 [2026-10-22 13:04:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4943 [2026-10-22 13:04:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3792 [2026-10-22 13:04:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5252 [2026-10-22 13:04:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3885 [2026-10-22 13:05:06][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.1681 [2026-10-22 13:05:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8700 [2026-10-22 13:05:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8579 [2026-10-22 13:05:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0209 [2026-10-22 13:05:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3153 [2026-10-22 13:05:24][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.1056 [2026-10-22 13:05:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.1661 [2026-10-22 13:05:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=3.9628 [2026-10-22 13:05:38][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,620 | MFU: 1.8% [2026-10-22 13:05:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2111 [2026-10-22 13:05:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0691 [2026-10-22 13:05:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0024 [2026-10-22 13:05:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5485 [2026-10-22 13:05:43][INFO][training.pretrain_engine] [Ph2|Step 760] loss=4.6945 | ppl=109.34 | aux=0.02505 | lr=2.96e-04 | gnorm=0.508 | tok/s=29,351 | MFU=7.0% | step=142900ms | tokens=1.599B | mem=4.2/32.7GB | cost=$62.36/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 19:55 (126.8h) | Phase3: SKIPPED | SFT: Oct 28 00:39 (131.6h) | DPO (Final): Oct 28 01:13 (132.1h) [2026-10-22 13:05:43][INFO][training.pretrain_engine] Starting global step 761 (accumulating 32 micro-batches)... [2026-10-22 13:05:43][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 13:06:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9228 [2026-10-22 13:06:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4330 [2026-10-22 13:06:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.9917 [2026-10-22 13:06:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9068 [2026-10-22 13:06:18][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.9690 [2026-10-22 13:06:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4431 [2026-10-22 13:06:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5281 [2026-10-22 13:06:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5248 [2026-10-22 13:06:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.1463 [2026-10-22 13:06:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.0652 [2026-10-22 13:06:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7165 [2026-10-22 13:06:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6083 [2026-10-22 13:06:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9650 [2026-10-22 13:06:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7808 [2026-10-22 13:06:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2739 [2026-10-22 13:06:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0575 [2026-10-22 13:07:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5879 [2026-10-22 13:07:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6755 [2026-10-22 13:07:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7835 [2026-10-22 13:07:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7080 [2026-10-22 13:07:29][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3396 [2026-10-22 13:07:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9815 [2026-10-22 13:07:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0191 [2026-10-22 13:07:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1664 [2026-10-22 13:07:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4839 [2026-10-22 13:07:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6806 [2026-10-22 13:07:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5461 [2026-10-22 13:07:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5269 [2026-10-22 13:08:01][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,610 | MFU: 1.8% [2026-10-22 13:08:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9347 [2026-10-22 13:08:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1164 [2026-10-22 13:08:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5976 [2026-10-22 13:08:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8491 [2026-10-22 13:08:06][INFO][training.pretrain_engine] [Ph2|Step 761] loss=4.7352 | ppl=113.89 | aux=0.02504 | lr=2.96e-04 | gnorm=0.520 | tok/s=29,353 | MFU=7.0% | step=142892ms | tokens=1.600B | mem=4.2/32.7GB | cost=$62.44/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 19:56 (126.8h) | Phase3: SKIPPED | SFT: Oct 28 00:40 (131.5h) | DPO (Final): Oct 28 01:14 (132.1h) [2026-10-22 13:08:06][INFO][training.pretrain_engine] Starting global step 762 (accumulating 32 micro-batches)... [2026-10-22 13:08:06][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 13:08:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9094 [2026-10-22 13:08:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1016 [2026-10-22 13:08:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8935 [2026-10-22 13:08:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9226 [2026-10-22 13:08:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8796 [2026-10-22 13:08:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9942 [2026-10-22 13:08:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6743 [2026-10-22 13:08:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9041 [2026-10-22 13:08:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6217 [2026-10-22 13:08:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4557 [2026-10-22 13:08:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7540 [2026-10-22 13:08:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7308 [2026-10-22 13:09:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9138 [2026-10-22 13:09:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9350 [2026-10-22 13:09:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6320 [2026-10-22 13:09:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8572 [2026-10-22 13:09:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5112 [2026-10-22 13:09:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8470 [2026-10-22 13:09:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.1332 [2026-10-22 13:09:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7010 [2026-10-22 13:09:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.0817 [2026-10-22 13:09:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2445 [2026-10-22 13:09:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.0971 [2026-10-22 13:09:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=3.9748 [2026-10-22 13:10:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=3.7672 [2026-10-22 13:10:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5299 [2026-10-22 13:10:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0611 [2026-10-22 13:10:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0167 [2026-10-22 13:10:23][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,604 | MFU: 1.8% [2026-10-22 13:10:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6784 [2026-10-22 13:10:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9358 [2026-10-22 13:10:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4660 [2026-10-22 13:10:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9730 [2026-10-22 13:10:29][INFO][training.pretrain_engine] [Ph2|Step 762] loss=4.6955 | ppl=109.46 | aux=0.02506 | lr=2.96e-04 | gnorm=0.566 | tok/s=29,354 | MFU=7.0% | step=142888ms | tokens=1.601B | mem=4.2/32.7GB | cost=$62.52/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 19:58 (126.8h) | Phase3: SKIPPED | SFT: Oct 28 00:42 (131.5h) | DPO (Final): Oct 28 01:16 (132.1h) [2026-10-22 13:10:29][INFO][training.pretrain_engine] Starting global step 763 (accumulating 32 micro-batches)... [2026-10-22 13:10:29][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 13:10:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3476 [2026-10-22 13:10:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.0923 [2026-10-22 13:10:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0464 [2026-10-22 13:10:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6828 [2026-10-22 13:11:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4355 [2026-10-22 13:11:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0519 [2026-10-22 13:11:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7090 [2026-10-22 13:11:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6541 [2026-10-22 13:11:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1928 [2026-10-22 13:11:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7721 [2026-10-22 13:11:22][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0406 [2026-10-22 13:11:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1868 [2026-10-22 13:11:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3268 [2026-10-22 13:11:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1139 [2026-10-22 13:11:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6919 [2026-10-22 13:11:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7619 [2026-10-22 13:11:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0943 [2026-10-22 13:11:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7626 [2026-10-22 13:11:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8667 [2026-10-22 13:11:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6774 [2026-10-22 13:12:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.0027 [2026-10-22 13:12:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.0874 [2026-10-22 13:12:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.0878 [2026-10-22 13:12:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2874 [2026-10-22 13:12:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9066 [2026-10-22 13:12:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8757 [2026-10-22 13:12:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0677 [2026-10-22 13:12:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6632 [2026-10-22 13:12:47][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,586 | MFU: 1.8% [2026-10-22 13:12:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=3.8682 [2026-10-22 13:12:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4551 [2026-10-22 13:12:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6214 [2026-10-22 13:12:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3973 [2026-10-22 13:12:52][INFO][training.pretrain_engine] [Ph2|Step 763] loss=4.6176 | ppl=101.26 | aux=0.02505 | lr=2.96e-04 | gnorm=0.629 | tok/s=29,354 | MFU=7.0% | step=142885ms | tokens=1.602B | mem=4.2/32.7GB | cost=$62.60/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 19:59 (126.8h) | Phase3: SKIPPED | SFT: Oct 28 00:43 (131.5h) | DPO (Final): Oct 28 01:17 (132.1h) [2026-10-22 13:12:52][INFO][training.pretrain_engine] Starting global step 764 (accumulating 32 micro-batches)... [2026-10-22 13:12:52][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 13:13:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8530 [2026-10-22 13:13:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8129 [2026-10-22 13:13:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7591 [2026-10-22 13:13:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7965 [2026-10-22 13:13:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1315 [2026-10-22 13:13:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.0093 [2026-10-22 13:13:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.1573 [2026-10-22 13:13:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2699 [2026-10-22 13:13:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0715 [2026-10-22 13:13:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0862 [2026-10-22 13:13:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0751 [2026-10-22 13:13:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7372 [2026-10-22 13:14:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6034 [2026-10-22 13:14:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3612 [2026-10-22 13:14:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4763 [2026-10-22 13:14:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4897 [2026-10-22 13:14:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4508 [2026-10-22 13:14:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5642 [2026-10-22 13:14:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5212 [2026-10-22 13:14:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5581 [2026-10-22 13:14:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3810 [2026-10-22 13:14:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2361 [2026-10-22 13:14:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6495 [2026-10-22 13:14:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5888 [2026-10-22 13:14:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8022 [2026-10-22 13:14:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0578 [2026-10-22 13:14:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8165 [2026-10-22 13:14:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8603 [2026-10-22 13:15:10][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,603 | MFU: 1.8% [2026-10-22 13:15:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0160 [2026-10-22 13:15:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0178 [2026-10-22 13:15:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0386 [2026-10-22 13:15:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9470 [2026-10-22 13:15:15][INFO][training.pretrain_engine] [Ph2|Step 764] loss=4.6141 | ppl=100.89 | aux=0.02507 | lr=2.96e-04 | gnorm=0.594 | tok/s=29,355 | MFU=7.0% | step=142884ms | tokens=1.603B | mem=4.2/32.7GB | cost=$62.68/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 20:01 (126.8h) | Phase3: SKIPPED | SFT: Oct 28 00:45 (131.5h) | DPO (Final): Oct 28 01:19 (132.1h) [2026-10-22 13:15:15][INFO][training.pretrain_engine] Starting global step 765 (accumulating 32 micro-batches)... [2026-10-22 13:15:15][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 13:15:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2089 [2026-10-22 13:15:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6791 [2026-10-22 13:15:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6732 [2026-10-22 13:15:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5142 [2026-10-22 13:15:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9581 [2026-10-22 13:15:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4936 [2026-10-22 13:15:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4347 [2026-10-22 13:15:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0621 [2026-10-22 13:16:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2180 [2026-10-22 13:16:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4243 [2026-10-22 13:16:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5191 [2026-10-22 13:16:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7045 [2026-10-22 13:16:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0925 [2026-10-22 13:16:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0329 [2026-10-22 13:16:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2729 [2026-10-22 13:16:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0977 [2026-10-22 13:16:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7764 [2026-10-22 13:16:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0204 [2026-10-22 13:16:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3124 [2026-10-22 13:16:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7308 [2026-10-22 13:17:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9869 [2026-10-22 13:17:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0654 [2026-10-22 13:17:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7940 [2026-10-22 13:17:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6498 [2026-10-22 13:17:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.1453 [2026-10-22 13:17:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=3.9702 [2026-10-22 13:17:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4719 [2026-10-22 13:17:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.1552 [2026-10-22 13:17:33][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,613 | MFU: 1.8% [2026-10-22 13:17:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=3.5293 [2026-10-22 13:17:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=3.9615 [2026-10-22 13:17:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4143 [2026-10-22 13:17:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7024 [2026-10-22 13:17:38][INFO][training.pretrain_engine] [Ph2|Step 765] loss=4.8548 | ppl=128.35 | aux=0.02505 | lr=2.96e-04 | gnorm=0.551 | tok/s=29,356 | MFU=7.0% | step=142878ms | tokens=1.604B | mem=4.2/32.7GB | cost=$62.77/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 20:02 (126.8h) | Phase3: SKIPPED | SFT: Oct 28 00:46 (131.5h) | DPO (Final): Oct 28 01:20 (132.1h) [2026-10-22 13:17:41][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step765 | Tokens: 1.604B | Spent: $62.77 | Remaining: $937.23 [2026-10-22 13:17:41][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step765 [2026-10-22 13:17:41][INFO][training.pretrain_engine] Starting global step 766 (accumulating 32 micro-batches)... [2026-10-22 13:17:41][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 13:17:42][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-22 13:17:42][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-22 13:17:42][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-22 13:17:42][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step765 to Phase-Technologies/forge-500m-pretrain (path: phase2_step765)... [2026-10-22 13:17:42][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-22 13:17:51][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-22 13:17:51][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step765 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-22 13:17:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1133 [2026-10-22 13:17:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1668 [2026-10-22 13:17:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.8548 [2026-10-22 13:17:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1349 [2026-10-22 13:18:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.6570 [2026-10-22 13:18:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.7733 [2026-10-22 13:18:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3993 [2026-10-22 13:18:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3652 [2026-10-22 13:18:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5710 [2026-10-22 13:18:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5334 [2026-10-22 13:18:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0066 [2026-10-22 13:18:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9086 [2026-10-22 13:18:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8291 [2026-10-22 13:18:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6010 [2026-10-22 13:18:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8122 [2026-10-22 13:18:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7500 [2026-10-22 13:19:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5064 [2026-10-22 13:19:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5447 [2026-10-22 13:19:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6124 [2026-10-22 13:19:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5527 [2026-10-22 13:19:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8765 [2026-10-22 13:19:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.1886 [2026-10-22 13:19:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7390 [2026-10-22 13:19:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5306 [2026-10-22 13:19:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9023 [2026-10-22 13:19:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7952 [2026-10-22 13:19:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5546 [2026-10-22 13:19:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8716 [2026-10-22 13:19:59][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,597 | MFU: 1.8% [2026-10-22 13:20:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.1673 [2026-10-22 13:20:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.0326 [2026-10-22 13:20:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6073 [2026-10-22 13:20:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7987 [2026-10-22 13:20:04][INFO][training.pretrain_engine] [Ph2|Step 766] loss=4.6304 | ppl=102.56 | aux=0.02506 | lr=2.96e-04 | gnorm=0.625 | tok/s=29,358 | MFU=7.0% | step=142868ms | tokens=1.605B | mem=4.2/32.7GB | cost=$62.85/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 20:04 (126.7h) | Phase3: SKIPPED | SFT: Oct 28 00:48 (131.5h) | DPO (Final): Oct 28 01:22 (132.0h) [2026-10-22 13:20:04][INFO][training.pretrain_engine] Starting global step 767 (accumulating 32 micro-batches)... [2026-10-22 13:20:04][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 13:20:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4793 [2026-10-22 13:20:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4253 [2026-10-22 13:20:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5030 [2026-10-22 13:20:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7108 [2026-10-22 13:20:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0804 [2026-10-22 13:20:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9485 [2026-10-22 13:20:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8439 [2026-10-22 13:20:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3336 [2026-10-22 13:20:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5401 [2026-10-22 13:20:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.1899 [2026-10-22 13:20:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2145 [2026-10-22 13:20:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4338 [2026-10-22 13:21:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8037 [2026-10-22 13:21:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7611 [2026-10-22 13:21:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1807 [2026-10-22 13:21:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0390 [2026-10-22 13:21:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.1898 [2026-10-22 13:21:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4964 [2026-10-22 13:21:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3078 [2026-10-22 13:21:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9972 [2026-10-22 13:21:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2697 [2026-10-22 13:21:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5585 [2026-10-22 13:21:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7285 [2026-10-22 13:21:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8589 [2026-10-22 13:22:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9157 [2026-10-22 13:22:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0444 [2026-10-22 13:22:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3172 [2026-10-22 13:22:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6746 [2026-10-22 13:22:22][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,620 | MFU: 1.8% [2026-10-22 13:22:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9180 [2026-10-22 13:22:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7848 [2026-10-22 13:22:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6848 [2026-10-22 13:22:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1717 [2026-10-22 13:22:27][INFO][training.pretrain_engine] [Ph2|Step 767] loss=4.6320 | ppl=102.72 | aux=0.02506 | lr=2.96e-04 | gnorm=0.715 | tok/s=29,357 | MFU=7.0% | step=142873ms | tokens=1.606B | mem=4.2/32.7GB | cost=$62.93/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 20:06 (126.7h) | Phase3: SKIPPED | SFT: Oct 28 00:50 (131.5h) | DPO (Final): Oct 28 01:24 (132.0h) [2026-10-22 13:22:27][INFO][training.pretrain_engine] Starting global step 768 (accumulating 32 micro-batches)... [2026-10-22 13:22:27][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 13:22:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5969 [2026-10-22 13:22:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4437 [2026-10-22 13:22:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5208 [2026-10-22 13:22:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7778 [2026-10-22 13:23:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4716 [2026-10-22 13:23:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2479 [2026-10-22 13:23:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2455 [2026-10-22 13:23:03][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4705 [2026-10-22 13:23:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6233 [2026-10-22 13:23:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3267 [2026-10-22 13:23:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9064 [2026-10-22 13:23:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=3.6780 [2026-10-22 13:23:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1213 [2026-10-22 13:23:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6888 [2026-10-22 13:23:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7819 [2026-10-22 13:23:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9612 [2026-10-22 13:23:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6931 [2026-10-22 13:23:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8611 [2026-10-22 13:23:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8487 [2026-10-22 13:23:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0810 [2026-10-22 13:24:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1674 [2026-10-22 13:24:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5477 [2026-10-22 13:24:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3499 [2026-10-22 13:24:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3960 [2026-10-22 13:24:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2338 [2026-10-22 13:24:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2378 [2026-10-22 13:24:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8926 [2026-10-22 13:24:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3802 [2026-10-22 13:24:44][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,610 | MFU: 1.8% [2026-10-22 13:24:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6284 [2026-10-22 13:24:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9278 [2026-10-22 13:24:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0851 [2026-10-22 13:24:50][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9717 [2026-10-22 13:24:50][INFO][training.pretrain_engine] [Ph2|Step 768] loss=4.6579 | ppl=105.42 | aux=0.02505 | lr=2.96e-04 | gnorm=0.793 | tok/s=29,357 | MFU=7.0% | step=142872ms | tokens=1.607B | mem=4.2/32.7GB | cost=$63.01/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 20:08 (126.7h) | Phase3: SKIPPED | SFT: Oct 28 00:51 (131.5h) | DPO (Final): Oct 28 01:25 (132.0h) [2026-10-22 13:24:50][INFO][training.pretrain_engine] Starting global step 769 (accumulating 32 micro-batches)... [2026-10-22 13:24:50][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 13:25:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6294 [2026-10-22 13:25:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9973 [2026-10-22 13:25:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1796 [2026-10-22 13:25:08][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2404 [2026-10-22 13:25:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.0763 [2026-10-22 13:25:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.0761 [2026-10-22 13:25:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2958 [2026-10-22 13:25:26][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.6062 [2026-10-22 13:25:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9148 [2026-10-22 13:25:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2675 [2026-10-22 13:25:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1655 [2026-10-22 13:25:44][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0872 [2026-10-22 13:26:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0208 [2026-10-22 13:26:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5145 [2026-10-22 13:26:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8521 [2026-10-22 13:26:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8747 [2026-10-22 13:26:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=3.3126 [2026-10-22 13:26:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4651 [2026-10-22 13:26:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6183 [2026-10-22 13:26:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4091 [2026-10-22 13:26:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=6.0159 [2026-10-22 13:26:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7146 [2026-10-22 13:26:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7311 [2026-10-22 13:26:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5479 [2026-10-22 13:26:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6228 [2026-10-22 13:26:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2451 [2026-10-22 13:26:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2196 [2026-10-22 13:26:55][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3000 [2026-10-22 13:27:08][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,590 | MFU: 1.8% [2026-10-22 13:27:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.0405 [2026-10-22 13:27:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.0410 [2026-10-22 13:27:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6764 [2026-10-22 13:27:13][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2831 [2026-10-22 13:27:13][INFO][training.pretrain_engine] [Ph2|Step 769] loss=4.7236 | ppl=112.57 | aux=0.02506 | lr=2.96e-04 | gnorm=0.750 | tok/s=29,356 | MFU=7.0% | step=142877ms | tokens=1.609B | mem=4.2/32.7GB | cost=$63.09/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 20:10 (126.7h) | Phase3: SKIPPED | SFT: Oct 28 00:53 (131.4h) | DPO (Final): Oct 28 01:28 (132.0h) [2026-10-22 13:27:13][INFO][training.pretrain_engine] Starting global step 770 (accumulating 32 micro-batches)... [2026-10-22 13:27:13][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 13:27:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6451 [2026-10-22 13:27:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7652 [2026-10-22 13:27:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5827 [2026-10-22 13:27:31][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7609 [2026-10-22 13:27:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6674 [2026-10-22 13:27:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1741 [2026-10-22 13:27:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8858 [2026-10-22 13:27:49][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0861 [2026-10-22 13:28:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.9954 [2026-10-22 13:28:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4025 [2026-10-22 13:28:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1002 [2026-10-22 13:28:07][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2799 [2026-10-22 13:28:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7563 [2026-10-22 13:28:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5226 [2026-10-22 13:28:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0101 [2026-10-22 13:28:25][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3600 [2026-10-22 13:28:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2162 [2026-10-22 13:28:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5095 [2026-10-22 13:28:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0377 [2026-10-22 13:28:43][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.2720 [2026-10-22 13:29:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8061 [2026-10-22 13:29:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5966 [2026-10-22 13:29:00][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5807 [2026-10-22 13:29:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5900 [2026-10-22 13:29:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2751 [2026-10-22 13:29:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4375 [2026-10-22 13:29:18][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4905 [2026-10-22 13:29:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7236 [2026-10-22 13:29:31][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,598 | MFU: 1.8% [2026-10-22 13:29:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8623 [2026-10-22 13:29:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8769 [2026-10-22 13:29:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8587 [2026-10-22 13:29:36][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6644 [2026-10-22 13:29:36][INFO][training.pretrain_engine] [Ph2|Step 770] loss=4.8362 | ppl=125.99 | aux=0.02505 | lr=2.96e-04 | gnorm=0.723 | tok/s=29,356 | MFU=7.0% | step=142880ms | tokens=1.610B | mem=4.2/32.7GB | cost=$63.18/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 20:12 (126.7h) | Phase3: SKIPPED | SFT: Oct 28 00:55 (131.4h) | DPO (Final): Oct 28 01:30 (132.0h) [2026-10-22 13:29:36][INFO][training.pretrain_engine] Starting global step 771 (accumulating 32 micro-batches)... [2026-10-22 13:29:36][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 13:29:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2118 [2026-10-22 13:29:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4296 [2026-10-22 13:29:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4443 [2026-10-22 13:29:54][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8826 [2026-10-22 13:30:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6828 [2026-10-22 13:30:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4266 [2026-10-22 13:30:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3656 [2026-10-22 13:30:12][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7757 [2026-10-22 13:30:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9311 [2026-10-22 13:30:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0205 [2026-10-22 13:30:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6450 [2026-10-22 13:30:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5455 [2026-10-22 13:30:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2096 [2026-10-22 13:30:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4419 [2026-10-22 13:30:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4152 [2026-10-22 13:30:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5675 [2026-10-22 13:31:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6448 [2026-10-22 13:31:05][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5123 [2026-10-22 13:31:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5864 [2026-10-22 13:31:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6456 [2026-10-22 13:31:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3003 [2026-10-22 13:31:23][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6608 [2026-10-22 13:31:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2632 [2026-10-22 13:31:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2426 [2026-10-22 13:31:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3796 [2026-10-22 13:31:41][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8172 [2026-10-22 13:31:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3978 [2026-10-22 13:31:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4661 [2026-10-22 13:31:54][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,601 | MFU: 1.8% [2026-10-22 13:32:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8772 [2026-10-22 13:32:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8416 [2026-10-22 13:32:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4088 [2026-10-22 13:32:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7361 [2026-10-22 13:32:00][INFO][training.pretrain_engine] [Ph2|Step 771] loss=4.7157 | ppl=111.69 | aux=0.02505 | lr=2.96e-04 | gnorm=0.789 | tok/s=29,355 | MFU=7.0% | step=142881ms | tokens=1.611B | mem=4.2/32.7GB | cost=$63.26/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 20:13 (126.7h) | Phase3: SKIPPED | SFT: Oct 28 00:57 (131.4h) | DPO (Final): Oct 28 01:31 (132.0h) [2026-10-22 13:32:00][INFO][training.pretrain_engine] Starting global step 772 (accumulating 32 micro-batches)... [2026-10-22 13:32:00][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 13:32:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6992 [2026-10-22 13:32:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7783 [2026-10-22 13:32:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8514 [2026-10-22 13:32:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4608 [2026-10-22 13:32:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8775 [2026-10-22 13:32:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8178 [2026-10-22 13:32:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8359 [2026-10-22 13:32:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0692 [2026-10-22 13:32:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6151 [2026-10-22 13:32:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9315 [2026-10-22 13:32:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2163 [2026-10-22 13:32:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8404 [2026-10-22 13:33:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0442 [2026-10-22 13:33:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8632 [2026-10-22 13:33:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.4718 [2026-10-22 13:33:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0570 [2026-10-22 13:33:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.0270 [2026-10-22 13:33:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0282 [2026-10-22 13:33:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.0172 [2026-10-22 13:33:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4490 [2026-10-22 13:33:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2421 [2026-10-22 13:33:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9759 [2026-10-22 13:33:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9532 [2026-10-22 13:33:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3026 [2026-10-22 13:34:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.1780 [2026-10-22 13:34:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5388 [2026-10-22 13:34:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7683 [2026-10-22 13:34:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7000 [2026-10-22 13:34:18][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,596 | MFU: 1.8% [2026-10-22 13:34:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2752 [2026-10-22 13:34:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7862 [2026-10-22 13:34:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5030 [2026-10-22 13:34:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9022 [2026-10-22 13:34:23][INFO][training.pretrain_engine] [Ph2|Step 772] loss=4.7538 | ppl=116.02 | aux=0.02506 | lr=2.96e-04 | gnorm=0.969 | tok/s=29,355 | MFU=7.0% | step=142883ms | tokens=1.612B | mem=4.2/32.7GB | cost=$63.34/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 20:15 (126.7h) | Phase3: SKIPPED | SFT: Oct 28 00:59 (131.4h) | DPO (Final): Oct 28 01:33 (132.0h) [2026-10-22 13:34:23][INFO][training.pretrain_engine] Starting global step 773 (accumulating 32 micro-batches)... [2026-10-22 13:34:23][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 13:34:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.1294 [2026-10-22 13:34:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.9060 [2026-10-22 13:34:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9999 [2026-10-22 13:34:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.0450 [2026-10-22 13:34:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6947 [2026-10-22 13:34:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0109 [2026-10-22 13:34:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2124 [2026-10-22 13:34:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2550 [2026-10-22 13:35:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.0687 [2026-10-22 13:35:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3828 [2026-10-22 13:35:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5964 [2026-10-22 13:35:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.1290 [2026-10-22 13:35:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3429 [2026-10-22 13:35:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5144 [2026-10-22 13:35:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4999 [2026-10-22 13:35:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4136 [2026-10-22 13:35:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3942 [2026-10-22 13:35:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4702 [2026-10-22 13:35:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8029 [2026-10-22 13:35:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=3.9816 [2026-10-22 13:36:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7290 [2026-10-22 13:36:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8144 [2026-10-22 13:36:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7747 [2026-10-22 13:36:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9711 [2026-10-22 13:36:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1840 [2026-10-22 13:36:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6074 [2026-10-22 13:36:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.1463 [2026-10-22 13:36:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7488 [2026-10-22 13:36:41][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,597 | MFU: 1.8% [2026-10-22 13:36:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6500 [2026-10-22 13:36:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5000 [2026-10-22 13:36:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.1915 [2026-10-22 13:36:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2281 [2026-10-22 13:36:46][INFO][training.pretrain_engine] [Ph2|Step 773] loss=4.6621 | ppl=105.86 | aux=0.02506 | lr=2.96e-04 | gnorm=0.965 | tok/s=29,354 | MFU=7.0% | step=142885ms | tokens=1.613B | mem=4.2/32.7GB | cost=$63.42/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 20:17 (126.7h) | Phase3: SKIPPED | SFT: Oct 28 01:01 (131.4h) | DPO (Final): Oct 28 01:35 (132.0h) [2026-10-22 13:36:46][INFO][training.pretrain_engine] Starting global step 774 (accumulating 32 micro-batches)... [2026-10-22 13:36:46][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 13:37:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6073 [2026-10-22 13:37:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6704 [2026-10-22 13:37:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9595 [2026-10-22 13:37:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9653 [2026-10-22 13:37:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6239 [2026-10-22 13:37:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0757 [2026-10-22 13:37:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8521 [2026-10-22 13:37:22][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7570 [2026-10-22 13:37:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8339 [2026-10-22 13:37:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9355 [2026-10-22 13:37:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7771 [2026-10-22 13:37:40][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9941 [2026-10-22 13:37:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6145 [2026-10-22 13:37:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8781 [2026-10-22 13:37:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9731 [2026-10-22 13:37:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6369 [2026-10-22 13:38:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7184 [2026-10-22 13:38:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6857 [2026-10-22 13:38:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9428 [2026-10-22 13:38:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5464 [2026-10-22 13:38:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2198 [2026-10-22 13:38:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0203 [2026-10-22 13:38:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3696 [2026-10-22 13:38:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1086 [2026-10-22 13:38:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2127 [2026-10-22 13:38:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4045 [2026-10-22 13:38:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5697 [2026-10-22 13:38:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2720 [2026-10-22 13:39:04][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,596 | MFU: 1.8% [2026-10-22 13:39:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2428 [2026-10-22 13:39:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0161 [2026-10-22 13:39:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1649 [2026-10-22 13:39:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.3140 [2026-10-22 13:39:09][INFO][training.pretrain_engine] [Ph2|Step 774] loss=4.8554 | ppl=128.43 | aux=0.02504 | lr=2.96e-04 | gnorm=0.949 | tok/s=29,353 | MFU=7.0% | step=142890ms | tokens=1.614B | mem=4.2/32.7GB | cost=$63.50/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 20:19 (126.7h) | Phase3: SKIPPED | SFT: Oct 28 01:03 (131.4h) | DPO (Final): Oct 28 01:37 (132.0h) [2026-10-22 13:39:12][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step774 | Tokens: 1.614B | Spent: $63.51 | Remaining: $936.49 [2026-10-22 13:39:12][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step774 [2026-10-22 13:39:12][INFO][training.pretrain_engine] Starting global step 775 (accumulating 32 micro-batches)... [2026-10-22 13:39:12][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 13:39:13][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-22 13:39:13][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-22 13:39:13][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-22 13:39:13][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step774 to Phase-Technologies/forge-500m-pretrain (path: phase2_step774)... [2026-10-22 13:39:13][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-22 13:39:22][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-22 13:39:22][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step774 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-22 13:39:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6526 [2026-10-22 13:39:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0451 [2026-10-22 13:39:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4779 [2026-10-22 13:39:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0050 [2026-10-22 13:39:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6447 [2026-10-22 13:39:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4784 [2026-10-22 13:39:45][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4813 [2026-10-22 13:39:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5407 [2026-10-22 13:40:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8912 [2026-10-22 13:40:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3382 [2026-10-22 13:40:03][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4794 [2026-10-22 13:40:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4370 [2026-10-22 13:40:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0723 [2026-10-22 13:40:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3233 [2026-10-22 13:40:21][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.1987 [2026-10-22 13:40:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5477 [2026-10-22 13:40:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5327 [2026-10-22 13:40:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2424 [2026-10-22 13:40:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2837 [2026-10-22 13:40:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1314 [2026-10-22 13:40:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0785 [2026-10-22 13:40:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9664 [2026-10-22 13:40:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7555 [2026-10-22 13:40:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8638 [2026-10-22 13:41:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5625 [2026-10-22 13:41:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4970 [2026-10-22 13:41:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7313 [2026-10-22 13:41:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8491 [2026-10-22 13:41:31][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,583 | MFU: 1.8% [2026-10-22 13:41:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7219 [2026-10-22 13:41:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7764 [2026-10-22 13:41:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7343 [2026-10-22 13:41:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7586 [2026-10-22 13:41:35][INFO][training.pretrain_engine] [Ph2|Step 775] loss=4.7729 | ppl=118.26 | aux=0.02506 | lr=2.96e-04 | gnorm=0.688 | tok/s=29,354 | MFU=7.0% | step=142888ms | tokens=1.615B | mem=4.2/32.7GB | cost=$63.59/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 20:21 (126.7h) | Phase3: SKIPPED | SFT: Oct 28 01:05 (131.4h) | DPO (Final): Oct 28 01:39 (132.0h) [2026-10-22 13:41:35][INFO][training.pretrain_engine] Starting global step 776 (accumulating 32 micro-batches)... [2026-10-22 13:41:35][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 13:41:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6983 [2026-10-22 13:41:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8489 [2026-10-22 13:41:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8215 [2026-10-22 13:41:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9093 [2026-10-22 13:42:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6774 [2026-10-22 13:42:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5659 [2026-10-22 13:42:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5496 [2026-10-22 13:42:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4701 [2026-10-22 13:42:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9747 [2026-10-22 13:42:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6032 [2026-10-22 13:42:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4955 [2026-10-22 13:42:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8919 [2026-10-22 13:42:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1909 [2026-10-22 13:42:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7116 [2026-10-22 13:42:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8218 [2026-10-22 13:42:47][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5325 [2026-10-22 13:43:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.5741 [2026-10-22 13:43:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6359 [2026-10-22 13:43:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0533 [2026-10-22 13:43:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0896 [2026-10-22 13:43:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1932 [2026-10-22 13:43:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.1079 [2026-10-22 13:43:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.1719 [2026-10-22 13:43:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6271 [2026-10-22 13:43:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7507 [2026-10-22 13:43:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4597 [2026-10-22 13:43:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6696 [2026-10-22 13:43:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7961 [2026-10-22 13:43:53][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,607 | MFU: 1.8% [2026-10-22 13:43:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5528 [2026-10-22 13:43:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8363 [2026-10-22 13:43:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5695 [2026-10-22 13:43:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5873 [2026-10-22 13:43:58][INFO][training.pretrain_engine] [Ph2|Step 776] loss=4.7920 | ppl=120.54 | aux=0.02506 | lr=2.96e-04 | gnorm=0.652 | tok/s=29,353 | MFU=7.0% | step=142890ms | tokens=1.616B | mem=4.2/32.7GB | cost=$63.67/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 20:23 (126.7h) | Phase3: SKIPPED | SFT: Oct 28 01:07 (131.4h) | DPO (Final): Oct 28 01:41 (132.0h) [2026-10-22 13:43:58][INFO][training.pretrain_engine] Starting global step 777 (accumulating 32 micro-batches)... [2026-10-22 13:43:58][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 13:44:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1282 [2026-10-22 13:44:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4960 [2026-10-22 13:44:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8849 [2026-10-22 13:44:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9209 [2026-10-22 13:44:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1663 [2026-10-22 13:44:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1102 [2026-10-22 13:44:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7093 [2026-10-22 13:44:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4574 [2026-10-22 13:44:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6219 [2026-10-22 13:44:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5675 [2026-10-22 13:44:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7771 [2026-10-22 13:44:52][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4590 [2026-10-22 13:45:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9160 [2026-10-22 13:45:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8084 [2026-10-22 13:45:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0084 [2026-10-22 13:45:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4777 [2026-10-22 13:45:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5322 [2026-10-22 13:45:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9759 [2026-10-22 13:45:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6515 [2026-10-22 13:45:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7005 [2026-10-22 13:45:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=3.9899 [2026-10-22 13:45:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6705 [2026-10-22 13:45:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.1007 [2026-10-22 13:45:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4982 [2026-10-22 13:46:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8555 [2026-10-22 13:46:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7911 [2026-10-22 13:46:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7248 [2026-10-22 13:46:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0533 [2026-10-22 13:46:16][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,601 | MFU: 1.8% [2026-10-22 13:46:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5162 [2026-10-22 13:46:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7853 [2026-10-22 13:46:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9856 [2026-10-22 13:46:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5549 [2026-10-22 13:46:21][INFO][training.pretrain_engine] [Ph2|Step 777] loss=4.7144 | ppl=111.54 | aux=0.02505 | lr=2.96e-04 | gnorm=0.586 | tok/s=29,353 | MFU=7.0% | step=142890ms | tokens=1.617B | mem=4.2/32.7GB | cost=$63.75/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 20:25 (126.6h) | Phase3: SKIPPED | SFT: Oct 28 01:09 (131.4h) | DPO (Final): Oct 28 01:43 (131.9h) [2026-10-22 13:46:21][INFO][training.pretrain_engine] Starting global step 778 (accumulating 32 micro-batches)... [2026-10-22 13:46:21][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 13:46:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2294 [2026-10-22 13:46:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0075 [2026-10-22 13:46:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9085 [2026-10-22 13:46:39][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1765 [2026-10-22 13:46:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7843 [2026-10-22 13:46:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1257 [2026-10-22 13:46:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9326 [2026-10-22 13:46:57][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7486 [2026-10-22 13:47:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3028 [2026-10-22 13:47:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9042 [2026-10-22 13:47:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9456 [2026-10-22 13:47:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1518 [2026-10-22 13:47:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8373 [2026-10-22 13:47:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1961 [2026-10-22 13:47:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0444 [2026-10-22 13:47:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2262 [2026-10-22 13:47:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8292 [2026-10-22 13:47:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8578 [2026-10-22 13:47:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2685 [2026-10-22 13:47:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1685 [2026-10-22 13:48:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8156 [2026-10-22 13:48:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=3.8642 [2026-10-22 13:48:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8455 [2026-10-22 13:48:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6953 [2026-10-22 13:48:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9906 [2026-10-22 13:48:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8599 [2026-10-22 13:48:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0931 [2026-10-22 13:48:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1435 [2026-10-22 13:48:39][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,617 | MFU: 1.8% [2026-10-22 13:48:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8687 [2026-10-22 13:48:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9831 [2026-10-22 13:48:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1163 [2026-10-22 13:48:44][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5688 [2026-10-22 13:48:44][INFO][training.pretrain_engine] [Ph2|Step 778] loss=4.7822 | ppl=119.37 | aux=0.02505 | lr=2.96e-04 | gnorm=0.629 | tok/s=29,353 | MFU=7.0% | step=142891ms | tokens=1.618B | mem=4.2/32.7GB | cost=$63.83/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 20:27 (126.6h) | Phase3: SKIPPED | SFT: Oct 28 01:10 (131.4h) | DPO (Final): Oct 28 01:44 (131.9h) [2026-10-22 13:48:44][INFO][training.pretrain_engine] Starting global step 779 (accumulating 32 micro-batches)... [2026-10-22 13:48:44][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 13:49:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6183 [2026-10-22 13:49:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9356 [2026-10-22 13:49:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8228 [2026-10-22 13:49:02][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8508 [2026-10-22 13:49:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7454 [2026-10-22 13:49:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7191 [2026-10-22 13:49:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6616 [2026-10-22 13:49:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.8305 [2026-10-22 13:49:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8866 [2026-10-22 13:49:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3294 [2026-10-22 13:49:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5437 [2026-10-22 13:49:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5934 [2026-10-22 13:49:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6935 [2026-10-22 13:49:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3384 [2026-10-22 13:49:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4725 [2026-10-22 13:49:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7667 [2026-10-22 13:50:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7718 [2026-10-22 13:50:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8849 [2026-10-22 13:50:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5175 [2026-10-22 13:50:13][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9749 [2026-10-22 13:50:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7103 [2026-10-22 13:50:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8175 [2026-10-22 13:50:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3682 [2026-10-22 13:50:31][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6088 [2026-10-22 13:50:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4201 [2026-10-22 13:50:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7793 [2026-10-22 13:50:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.4751 [2026-10-22 13:50:49][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0388 [2026-10-22 13:51:02][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,594 | MFU: 1.8% [2026-10-22 13:51:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4859 [2026-10-22 13:51:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8527 [2026-10-22 13:51:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6856 [2026-10-22 13:51:07][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1007 [2026-10-22 13:51:07][INFO][training.pretrain_engine] [Ph2|Step 779] loss=4.6172 | ppl=101.21 | aux=0.02505 | lr=2.96e-04 | gnorm=0.598 | tok/s=29,353 | MFU=7.0% | step=142890ms | tokens=1.619B | mem=4.2/32.7GB | cost=$63.91/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 20:28 (126.6h) | Phase3: SKIPPED | SFT: Oct 28 01:12 (131.4h) | DPO (Final): Oct 28 01:46 (131.9h) [2026-10-22 13:51:07][INFO][training.pretrain_engine] Starting global step 780 (accumulating 32 micro-batches)... [2026-10-22 13:51:07][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 13:51:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2415 [2026-10-22 13:51:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4071 [2026-10-22 13:51:24][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6056 [2026-10-22 13:51:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7329 [2026-10-22 13:51:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7871 [2026-10-22 13:51:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6387 [2026-10-22 13:51:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7844 [2026-10-22 13:51:42][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2107 [2026-10-22 13:52:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0385 [2026-10-22 13:52:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8298 [2026-10-22 13:52:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7059 [2026-10-22 13:52:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5205 [2026-10-22 13:52:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9606 [2026-10-22 13:52:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0257 [2026-10-22 13:52:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8353 [2026-10-22 13:52:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5319 [2026-10-22 13:52:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3073 [2026-10-22 13:52:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.1071 [2026-10-22 13:52:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=3.5346 [2026-10-22 13:52:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.2314 [2026-10-22 13:52:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9944 [2026-10-22 13:52:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8238 [2026-10-22 13:52:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9192 [2026-10-22 13:52:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2305 [2026-10-22 13:53:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1124 [2026-10-22 13:53:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.1909 [2026-10-22 13:53:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5510 [2026-10-22 13:53:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3670 [2026-10-22 13:53:25][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,589 | MFU: 1.8% [2026-10-22 13:53:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1334 [2026-10-22 13:53:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8773 [2026-10-22 13:53:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4831 [2026-10-22 13:53:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0295 [2026-10-22 13:53:29][INFO][training.pretrain_engine] [Ph2|Step 780] loss=4.6807 | ppl=107.84 | aux=0.02506 | lr=2.96e-04 | gnorm=0.590 | tok/s=29,354 | MFU=7.0% | step=142886ms | tokens=1.620B | mem=4.2/32.7GB | cost=$64.00/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 20:30 (126.6h) | Phase3: SKIPPED | SFT: Oct 28 01:14 (131.3h) | DPO (Final): Oct 28 01:48 (131.9h) [2026-10-22 13:53:29][INFO][training.pretrain_engine] Starting global step 781 (accumulating 32 micro-batches)... [2026-10-22 13:53:29][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 13:53:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6220 [2026-10-22 13:53:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3080 [2026-10-22 13:53:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.1580 [2026-10-22 13:53:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4754 [2026-10-22 13:54:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8533 [2026-10-22 13:54:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6415 [2026-10-22 13:54:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8423 [2026-10-22 13:54:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7079 [2026-10-22 13:54:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4341 [2026-10-22 13:54:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8503 [2026-10-22 13:54:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1854 [2026-10-22 13:54:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8911 [2026-10-22 13:54:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.3616 [2026-10-22 13:54:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2741 [2026-10-22 13:54:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1401 [2026-10-22 13:54:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0474 [2026-10-22 13:54:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1150 [2026-10-22 13:54:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9212 [2026-10-22 13:54:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1036 [2026-10-22 13:54:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6782 [2026-10-22 13:55:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8848 [2026-10-22 13:55:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5633 [2026-10-22 13:55:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0062 [2026-10-22 13:55:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9948 [2026-10-22 13:55:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5507 [2026-10-22 13:55:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8613 [2026-10-22 13:55:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3692 [2026-10-22 13:55:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5339 [2026-10-22 13:55:47][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,610 | MFU: 1.8% [2026-10-22 13:55:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2863 [2026-10-22 13:55:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2292 [2026-10-22 13:55:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8727 [2026-10-22 13:55:52][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4800 [2026-10-22 13:55:52][INFO][training.pretrain_engine] [Ph2|Step 781] loss=4.7960 | ppl=121.03 | aux=0.02506 | lr=2.96e-04 | gnorm=0.586 | tok/s=29,354 | MFU=7.0% | step=142887ms | tokens=1.621B | mem=4.2/32.7GB | cost=$64.08/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 20:32 (126.6h) | Phase3: SKIPPED | SFT: Oct 28 01:16 (131.3h) | DPO (Final): Oct 28 01:50 (131.9h) [2026-10-22 13:55:52][INFO][training.pretrain_engine] Starting global step 782 (accumulating 32 micro-batches)... [2026-10-22 13:55:52][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 13:56:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6947 [2026-10-22 13:56:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3356 [2026-10-22 13:56:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3958 [2026-10-22 13:56:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5672 [2026-10-22 13:56:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3592 [2026-10-22 13:56:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5140 [2026-10-22 13:56:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1365 [2026-10-22 13:56:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3223 [2026-10-22 13:56:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1053 [2026-10-22 13:56:45][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0444 [2026-10-22 13:56:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3050 [2026-10-22 13:56:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7535 [2026-10-22 13:57:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8018 [2026-10-22 13:57:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8961 [2026-10-22 13:57:03][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4989 [2026-10-22 13:57:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7298 [2026-10-22 13:57:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7479 [2026-10-22 13:57:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7453 [2026-10-22 13:57:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8250 [2026-10-22 13:57:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6679 [2026-10-22 13:57:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1955 [2026-10-22 13:57:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0113 [2026-10-22 13:57:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2142 [2026-10-22 13:57:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2141 [2026-10-22 13:57:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9594 [2026-10-22 13:57:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7781 [2026-10-22 13:57:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=3.5418 [2026-10-22 13:57:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4511 [2026-10-22 13:58:10][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,594 | MFU: 1.8% [2026-10-22 13:58:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8235 [2026-10-22 13:58:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7725 [2026-10-22 13:58:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9941 [2026-10-22 13:58:15][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7099 [2026-10-22 13:58:15][INFO][training.pretrain_engine] [Ph2|Step 782] loss=4.7137 | ppl=111.46 | aux=0.02505 | lr=2.96e-04 | gnorm=0.531 | tok/s=29,353 | MFU=7.0% | step=142889ms | tokens=1.622B | mem=4.2/32.7GB | cost=$64.16/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 20:34 (126.6h) | Phase3: SKIPPED | SFT: Oct 28 01:17 (131.3h) | DPO (Final): Oct 28 01:52 (131.9h) [2026-10-22 13:58:15][INFO][training.pretrain_engine] Starting global step 783 (accumulating 32 micro-batches)... [2026-10-22 13:58:15][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 13:58:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9019 [2026-10-22 13:58:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3605 [2026-10-22 13:58:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9674 [2026-10-22 13:58:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8460 [2026-10-22 13:58:50][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.6401 [2026-10-22 13:58:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.9755 [2026-10-22 13:58:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4167 [2026-10-22 13:58:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4307 [2026-10-22 13:59:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3378 [2026-10-22 13:59:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1545 [2026-10-22 13:59:08][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.5021 [2026-10-22 13:59:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0220 [2026-10-22 13:59:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0349 [2026-10-22 13:59:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9071 [2026-10-22 13:59:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8284 [2026-10-22 13:59:26][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8611 [2026-10-22 13:59:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.1358 [2026-10-22 13:59:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4182 [2026-10-22 13:59:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3126 [2026-10-22 13:59:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4967 [2026-10-22 14:00:01][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5209 [2026-10-22 14:00:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6107 [2026-10-22 14:00:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7281 [2026-10-22 14:00:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.0757 [2026-10-22 14:00:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1907 [2026-10-22 14:00:19][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9890 [2026-10-22 14:00:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9125 [2026-10-22 14:00:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1064 [2026-10-22 14:00:33][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,595 | MFU: 1.8% [2026-10-22 14:00:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4438 [2026-10-22 14:00:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8958 [2026-10-22 14:00:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6511 [2026-10-22 14:00:38][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4449 [2026-10-22 14:00:38][INFO][training.pretrain_engine] [Ph2|Step 783] loss=4.7235 | ppl=112.56 | aux=0.02504 | lr=2.96e-04 | gnorm=0.531 | tok/s=29,354 | MFU=7.0% | step=142885ms | tokens=1.623B | mem=4.2/32.7GB | cost=$64.24/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 20:35 (126.6h) | Phase3: SKIPPED | SFT: Oct 28 01:19 (131.3h) | DPO (Final): Oct 28 01:53 (131.9h) [2026-10-22 14:00:41][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step783 | Tokens: 1.623B | Spent: $64.24 | Remaining: $935.76 [2026-10-22 14:00:41][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step783 [2026-10-22 14:00:41][INFO][training.pretrain_engine] Starting global step 784 (accumulating 32 micro-batches)... [2026-10-22 14:00:41][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 14:00:41][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-22 14:00:41][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-22 14:00:41][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-22 14:00:41][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step783 to Phase-Technologies/forge-500m-pretrain (path: phase2_step783)... [2026-10-22 14:00:42][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-22 14:00:51][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-22 14:00:51][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step783 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-22 14:00:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8318 [2026-10-22 14:00:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7896 [2026-10-22 14:00:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6489 [2026-10-22 14:00:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3897 [2026-10-22 14:01:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3829 [2026-10-22 14:01:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2712 [2026-10-22 14:01:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.0429 [2026-10-22 14:01:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6780 [2026-10-22 14:01:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4015 [2026-10-22 14:01:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4905 [2026-10-22 14:01:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3120 [2026-10-22 14:01:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4583 [2026-10-22 14:01:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5020 [2026-10-22 14:01:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.0605 [2026-10-22 14:01:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4713 [2026-10-22 14:01:53][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2352 [2026-10-22 14:02:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7924 [2026-10-22 14:02:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9712 [2026-10-22 14:02:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8299 [2026-10-22 14:02:11][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9861 [2026-10-22 14:02:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0945 [2026-10-22 14:02:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8494 [2026-10-22 14:02:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7539 [2026-10-22 14:02:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7414 [2026-10-22 14:02:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9406 [2026-10-22 14:02:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0252 [2026-10-22 14:02:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1707 [2026-10-22 14:02:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0875 [2026-10-22 14:02:59][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,595 | MFU: 1.8% [2026-10-22 14:03:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9801 [2026-10-22 14:03:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9101 [2026-10-22 14:03:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0041 [2026-10-22 14:03:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4035 [2026-10-22 14:03:04][INFO][training.pretrain_engine] [Ph2|Step 784] loss=4.6983 | ppl=109.76 | aux=0.02505 | lr=2.96e-04 | gnorm=0.516 | tok/s=29,355 | MFU=7.0% | step=142883ms | tokens=1.624B | mem=4.2/32.7GB | cost=$64.33/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 20:37 (126.6h) | Phase3: SKIPPED | SFT: Oct 28 01:21 (131.3h) | DPO (Final): Oct 28 01:55 (131.9h) [2026-10-22 14:03:04][INFO][training.pretrain_engine] Starting global step 785 (accumulating 32 micro-batches)... [2026-10-22 14:03:04][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 14:03:21][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8669 [2026-10-22 14:03:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1377 [2026-10-22 14:03:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8427 [2026-10-22 14:03:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8101 [2026-10-22 14:03:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3531 [2026-10-22 14:03:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7172 [2026-10-22 14:03:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2366 [2026-10-22 14:03:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.0508 [2026-10-22 14:03:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4513 [2026-10-22 14:03:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4717 [2026-10-22 14:03:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6455 [2026-10-22 14:03:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4514 [2026-10-22 14:04:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6534 [2026-10-22 14:04:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2137 [2026-10-22 14:04:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1228 [2026-10-22 14:04:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4308 [2026-10-22 14:04:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1206 [2026-10-22 14:04:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0583 [2026-10-22 14:04:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9028 [2026-10-22 14:04:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7111 [2026-10-22 14:04:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5201 [2026-10-22 14:04:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2865 [2026-10-22 14:04:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.6011 [2026-10-22 14:04:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0680 [2026-10-22 14:05:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1356 [2026-10-22 14:05:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5325 [2026-10-22 14:05:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9110 [2026-10-22 14:05:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5029 [2026-10-22 14:05:22][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,615 | MFU: 1.8% [2026-10-22 14:05:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4238 [2026-10-22 14:05:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4554 [2026-10-22 14:05:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3684 [2026-10-22 14:05:27][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2387 [2026-10-22 14:05:27][INFO][training.pretrain_engine] [Ph2|Step 785] loss=4.6952 | ppl=109.42 | aux=0.02506 | lr=2.96e-04 | gnorm=0.512 | tok/s=29,353 | MFU=7.0% | step=142890ms | tokens=1.625B | mem=4.2/32.7GB | cost=$64.41/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 20:39 (126.6h) | Phase3: SKIPPED | SFT: Oct 28 01:23 (131.3h) | DPO (Final): Oct 28 01:57 (131.9h) [2026-10-22 14:05:27][INFO][training.pretrain_engine] Starting global step 786 (accumulating 32 micro-batches)... [2026-10-22 14:05:27][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 14:05:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7768 [2026-10-22 14:05:44][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5555 [2026-10-22 14:05:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4698 [2026-10-22 14:05:45][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5639 [2026-10-22 14:06:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5196 [2026-10-22 14:06:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9354 [2026-10-22 14:06:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2274 [2026-10-22 14:06:02][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7199 [2026-10-22 14:06:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5592 [2026-10-22 14:06:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1029 [2026-10-22 14:06:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9916 [2026-10-22 14:06:20][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3321 [2026-10-22 14:06:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7499 [2026-10-22 14:06:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1383 [2026-10-22 14:06:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9729 [2026-10-22 14:06:38][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9120 [2026-10-22 14:06:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6853 [2026-10-22 14:06:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1897 [2026-10-22 14:06:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4640 [2026-10-22 14:06:56][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.2254 [2026-10-22 14:07:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7444 [2026-10-22 14:07:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0765 [2026-10-22 14:07:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6803 [2026-10-22 14:07:14][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9901 [2026-10-22 14:07:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2884 [2026-10-22 14:07:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5975 [2026-10-22 14:07:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8045 [2026-10-22 14:07:32][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8937 [2026-10-22 14:07:44][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,608 | MFU: 1.8% [2026-10-22 14:07:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3849 [2026-10-22 14:07:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0568 [2026-10-22 14:07:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4761 [2026-10-22 14:07:49][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7747 [2026-10-22 14:07:49][INFO][training.pretrain_engine] [Ph2|Step 786] loss=4.7312 | ppl=113.43 | aux=0.02505 | lr=2.96e-04 | gnorm=0.512 | tok/s=29,354 | MFU=7.0% | step=142887ms | tokens=1.626B | mem=4.2/32.7GB | cost=$64.49/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 20:41 (126.6h) | Phase3: SKIPPED | SFT: Oct 28 01:25 (131.3h) | DPO (Final): Oct 28 01:59 (131.9h) [2026-10-22 14:07:49][INFO][training.pretrain_engine] Starting global step 787 (accumulating 32 micro-batches)... [2026-10-22 14:07:49][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 14:08:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9716 [2026-10-22 14:08:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7142 [2026-10-22 14:08:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0970 [2026-10-22 14:08:07][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9821 [2026-10-22 14:08:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0453 [2026-10-22 14:08:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8839 [2026-10-22 14:08:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8534 [2026-10-22 14:08:25][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1199 [2026-10-22 14:08:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5101 [2026-10-22 14:08:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7021 [2026-10-22 14:08:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4838 [2026-10-22 14:08:43][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7139 [2026-10-22 14:09:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7066 [2026-10-22 14:09:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7610 [2026-10-22 14:09:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7080 [2026-10-22 14:09:01][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4066 [2026-10-22 14:09:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3150 [2026-10-22 14:09:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.0324 [2026-10-22 14:09:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3642 [2026-10-22 14:09:19][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.2255 [2026-10-22 14:09:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1847 [2026-10-22 14:09:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0158 [2026-10-22 14:09:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9832 [2026-10-22 14:09:37][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9519 [2026-10-22 14:09:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2527 [2026-10-22 14:09:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6574 [2026-10-22 14:09:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=3.7544 [2026-10-22 14:09:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2833 [2026-10-22 14:10:08][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,578 | MFU: 1.8% [2026-10-22 14:10:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8456 [2026-10-22 14:10:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9907 [2026-10-22 14:10:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5778 [2026-10-22 14:10:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7497 [2026-10-22 14:10:12][INFO][training.pretrain_engine] [Ph2|Step 787] loss=4.7138 | ppl=111.47 | aux=0.02505 | lr=2.96e-04 | gnorm=0.494 | tok/s=29,353 | MFU=7.0% | step=142890ms | tokens=1.627B | mem=4.2/32.7GB | cost=$64.57/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 20:43 (126.5h) | Phase3: SKIPPED | SFT: Oct 28 01:27 (131.3h) | DPO (Final): Oct 28 02:01 (131.8h) [2026-10-22 14:10:12][INFO][training.pretrain_engine] Starting global step 788 (accumulating 32 micro-batches)... [2026-10-22 14:10:12][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 14:10:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7453 [2026-10-22 14:10:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5423 [2026-10-22 14:10:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.9318 [2026-10-22 14:10:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.9983 [2026-10-22 14:10:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7514 [2026-10-22 14:10:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4500 [2026-10-22 14:10:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3960 [2026-10-22 14:10:48][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3982 [2026-10-22 14:11:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.0479 [2026-10-22 14:11:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=3.8824 [2026-10-22 14:11:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5781 [2026-10-22 14:11:06][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.1919 [2026-10-22 14:11:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7921 [2026-10-22 14:11:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8056 [2026-10-22 14:11:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8016 [2026-10-22 14:11:24][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0144 [2026-10-22 14:11:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0963 [2026-10-22 14:11:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2096 [2026-10-22 14:11:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6977 [2026-10-22 14:11:42][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.6927 [2026-10-22 14:11:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3024 [2026-10-22 14:11:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3387 [2026-10-22 14:11:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7629 [2026-10-22 14:11:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=3.9442 [2026-10-22 14:12:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0245 [2026-10-22 14:12:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1146 [2026-10-22 14:12:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2831 [2026-10-22 14:12:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=3.7014 [2026-10-22 14:12:30][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,602 | MFU: 1.8% [2026-10-22 14:12:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.5211 [2026-10-22 14:12:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1044 [2026-10-22 14:12:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2376 [2026-10-22 14:12:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8698 [2026-10-22 14:12:35][INFO][training.pretrain_engine] [Ph2|Step 788] loss=4.6726 | ppl=106.98 | aux=0.02505 | lr=2.96e-04 | gnorm=0.482 | tok/s=29,354 | MFU=7.0% | step=142886ms | tokens=1.628B | mem=4.2/32.7GB | cost=$64.65/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 20:44 (126.5h) | Phase3: SKIPPED | SFT: Oct 28 01:28 (131.3h) | DPO (Final): Oct 28 02:02 (131.8h) [2026-10-22 14:12:35][INFO][training.pretrain_engine] Starting global step 789 (accumulating 32 micro-batches)... [2026-10-22 14:12:35][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 14:12:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9214 [2026-10-22 14:12:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6723 [2026-10-22 14:12:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2816 [2026-10-22 14:12:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3217 [2026-10-22 14:13:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6709 [2026-10-22 14:13:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2813 [2026-10-22 14:13:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7304 [2026-10-22 14:13:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.8654 [2026-10-22 14:13:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0448 [2026-10-22 14:13:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9972 [2026-10-22 14:13:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9592 [2026-10-22 14:13:29][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8694 [2026-10-22 14:13:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1115 [2026-10-22 14:13:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9202 [2026-10-22 14:13:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2792 [2026-10-22 14:13:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3627 [2026-10-22 14:14:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7479 [2026-10-22 14:14:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.1033 [2026-10-22 14:14:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4592 [2026-10-22 14:14:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6140 [2026-10-22 14:14:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8738 [2026-10-22 14:14:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9759 [2026-10-22 14:14:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8891 [2026-10-22 14:14:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5940 [2026-10-22 14:14:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9128 [2026-10-22 14:14:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3406 [2026-10-22 14:14:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4821 [2026-10-22 14:14:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6418 [2026-10-22 14:14:53][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,605 | MFU: 1.8% [2026-10-22 14:14:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5397 [2026-10-22 14:14:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3387 [2026-10-22 14:14:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7170 [2026-10-22 14:14:58][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7325 [2026-10-22 14:14:58][INFO][training.pretrain_engine] [Ph2|Step 789] loss=4.7283 | ppl=113.10 | aux=0.02506 | lr=2.96e-04 | gnorm=0.520 | tok/s=29,354 | MFU=7.0% | step=142887ms | tokens=1.629B | mem=4.2/32.7GB | cost=$64.73/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 20:46 (126.5h) | Phase3: SKIPPED | SFT: Oct 28 01:30 (131.3h) | DPO (Final): Oct 28 02:04 (131.8h) [2026-10-22 14:14:58][INFO][training.pretrain_engine] Starting global step 790 (accumulating 32 micro-batches)... [2026-10-22 14:14:58][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 14:15:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9454 [2026-10-22 14:15:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6339 [2026-10-22 14:15:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8265 [2026-10-22 14:15:16][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9357 [2026-10-22 14:15:33][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4561 [2026-10-22 14:15:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2591 [2026-10-22 14:15:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2570 [2026-10-22 14:15:34][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8499 [2026-10-22 14:15:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0018 [2026-10-22 14:15:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4264 [2026-10-22 14:15:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6582 [2026-10-22 14:15:51][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5683 [2026-10-22 14:16:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3061 [2026-10-22 14:16:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2453 [2026-10-22 14:16:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4033 [2026-10-22 14:16:09][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4711 [2026-10-22 14:16:26][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1328 [2026-10-22 14:16:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2783 [2026-10-22 14:16:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8244 [2026-10-22 14:16:27][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1769 [2026-10-22 14:16:44][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8892 [2026-10-22 14:16:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4885 [2026-10-22 14:16:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7847 [2026-10-22 14:16:45][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6607 [2026-10-22 14:17:02][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.0552 [2026-10-22 14:17:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0733 [2026-10-22 14:17:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9399 [2026-10-22 14:17:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6631 [2026-10-22 14:17:16][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,599 | MFU: 1.8% [2026-10-22 14:17:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5351 [2026-10-22 14:17:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4794 [2026-10-22 14:17:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6221 [2026-10-22 14:17:21][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2073 [2026-10-22 14:17:21][INFO][training.pretrain_engine] [Ph2|Step 790] loss=4.7903 | ppl=120.34 | aux=0.02505 | lr=2.96e-04 | gnorm=0.475 | tok/s=29,354 | MFU=7.0% | step=142885ms | tokens=1.631B | mem=4.2/32.7GB | cost=$64.82/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 20:48 (126.5h) | Phase3: SKIPPED | SFT: Oct 28 01:32 (131.2h) | DPO (Final): Oct 28 02:06 (131.8h) [2026-10-22 14:17:21][INFO][training.pretrain_engine] Starting global step 791 (accumulating 32 micro-batches)... [2026-10-22 14:17:21][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 14:17:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.1632 [2026-10-22 14:17:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5700 [2026-10-22 14:17:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8183 [2026-10-22 14:17:38][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5656 [2026-10-22 14:17:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.0828 [2026-10-22 14:17:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4227 [2026-10-22 14:17:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1263 [2026-10-22 14:17:56][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7625 [2026-10-22 14:18:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6696 [2026-10-22 14:18:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=3.5872 [2026-10-22 14:18:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=3.3513 [2026-10-22 14:18:14][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6107 [2026-10-22 14:18:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7230 [2026-10-22 14:18:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6459 [2026-10-22 14:18:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3082 [2026-10-22 14:18:32][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0016 [2026-10-22 14:18:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9851 [2026-10-22 14:18:49][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8994 [2026-10-22 14:18:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5328 [2026-10-22 14:18:50][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4385 [2026-10-22 14:19:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0025 [2026-10-22 14:19:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8008 [2026-10-22 14:19:07][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3295 [2026-10-22 14:19:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=3.8979 [2026-10-22 14:19:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=3.9557 [2026-10-22 14:19:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5951 [2026-10-22 14:19:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3091 [2026-10-22 14:19:25][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5043 [2026-10-22 14:19:38][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,607 | MFU: 1.8% [2026-10-22 14:19:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5153 [2026-10-22 14:19:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9460 [2026-10-22 14:19:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2508 [2026-10-22 14:19:43][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6347 [2026-10-22 14:19:43][INFO][training.pretrain_engine] [Ph2|Step 791] loss=4.5530 | ppl=94.91 | aux=0.02506 | lr=2.96e-04 | gnorm=0.523 | tok/s=29,354 | MFU=7.0% | step=142887ms | tokens=1.632B | mem=4.2/32.7GB | cost=$64.90/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 20:50 (126.5h) | Phase3: SKIPPED | SFT: Oct 28 01:33 (131.2h) | DPO (Final): Oct 28 02:08 (131.8h) [2026-10-22 14:19:43][INFO][training.pretrain_engine] Starting global step 792 (accumulating 32 micro-batches)... [2026-10-22 14:19:43][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 14:20:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1642 [2026-10-22 14:20:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0543 [2026-10-22 14:20:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9550 [2026-10-22 14:20:01][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3199 [2026-10-22 14:20:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0274 [2026-10-22 14:20:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7098 [2026-10-22 14:20:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0392 [2026-10-22 14:20:19][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1631 [2026-10-22 14:20:36][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1452 [2026-10-22 14:20:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7579 [2026-10-22 14:20:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9693 [2026-10-22 14:20:37][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2143 [2026-10-22 14:20:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8288 [2026-10-22 14:20:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9227 [2026-10-22 14:20:54][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2898 [2026-10-22 14:20:55][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8759 [2026-10-22 14:21:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2565 [2026-10-22 14:21:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7488 [2026-10-22 14:21:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.0731 [2026-10-22 14:21:12][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5001 [2026-10-22 14:21:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2412 [2026-10-22 14:21:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0968 [2026-10-22 14:21:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0990 [2026-10-22 14:21:30][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2134 [2026-10-22 14:21:47][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7879 [2026-10-22 14:21:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5036 [2026-10-22 14:21:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5430 [2026-10-22 14:21:48][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2333 [2026-10-22 14:22:01][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,617 | MFU: 1.8% [2026-10-22 14:22:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0339 [2026-10-22 14:22:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1005 [2026-10-22 14:22:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0655 [2026-10-22 14:22:06][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1360 [2026-10-22 14:22:06][INFO][training.pretrain_engine] [Ph2|Step 792] loss=4.8135 | ppl=123.17 | aux=0.02504 | lr=2.96e-04 | gnorm=0.459 | tok/s=29,354 | MFU=7.0% | step=142886ms | tokens=1.633B | mem=4.2/32.7GB | cost=$64.98/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 20:51 (126.5h) | Phase3: SKIPPED | SFT: Oct 28 01:35 (131.2h) | DPO (Final): Oct 28 02:09 (131.8h) [2026-10-22 14:22:09][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step792 | Tokens: 1.633B | Spent: $64.98 | Remaining: $935.02 [2026-10-22 14:22:09][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step792 [2026-10-22 14:22:09][INFO][training.pretrain_engine] Starting global step 793 (accumulating 32 micro-batches)... [2026-10-22 14:22:09][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 14:22:09][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-22 14:22:09][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-22 14:22:09][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-22 14:22:09][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step792 to Phase-Technologies/forge-500m-pretrain (path: phase2_step792)... [2026-10-22 14:22:09][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-22 14:22:18][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-22 14:22:18][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step792 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-22 14:22:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7597 [2026-10-22 14:22:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8558 [2026-10-22 14:22:23][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6486 [2026-10-22 14:22:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0367 [2026-10-22 14:22:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6519 [2026-10-22 14:22:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0225 [2026-10-22 14:22:41][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5973 [2026-10-22 14:22:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6289 [2026-10-22 14:22:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6318 [2026-10-22 14:22:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7657 [2026-10-22 14:22:59][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6105 [2026-10-22 14:23:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7437 [2026-10-22 14:23:16][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5920 [2026-10-22 14:23:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5629 [2026-10-22 14:23:17][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.1220 [2026-10-22 14:23:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5992 [2026-10-22 14:23:34][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4742 [2026-10-22 14:23:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5803 [2026-10-22 14:23:35][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0335 [2026-10-22 14:23:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.4986 [2026-10-22 14:23:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9511 [2026-10-22 14:23:52][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8872 [2026-10-22 14:23:53][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7737 [2026-10-22 14:23:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3433 [2026-10-22 14:24:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6367 [2026-10-22 14:24:10][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5731 [2026-10-22 14:24:11][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4601 [2026-10-22 14:24:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2197 [2026-10-22 14:24:27][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,580 | MFU: 1.8% [2026-10-22 14:24:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2698 [2026-10-22 14:24:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5329 [2026-10-22 14:24:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.4307 [2026-10-22 14:24:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1412 [2026-10-22 14:24:32][INFO][training.pretrain_engine] [Ph2|Step 793] loss=4.6993 | ppl=109.87 | aux=0.02505 | lr=2.96e-04 | gnorm=0.500 | tok/s=29,355 | MFU=7.0% | step=142884ms | tokens=1.634B | mem=4.2/32.7GB | cost=$65.06/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 20:53 (126.5h) | Phase3: SKIPPED | SFT: Oct 28 01:37 (131.2h) | DPO (Final): Oct 28 02:11 (131.8h) [2026-10-22 14:24:32][INFO][training.pretrain_engine] Starting global step 794 (accumulating 32 micro-batches)... [2026-10-22 14:24:32][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 14:24:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9682 [2026-10-22 14:24:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3295 [2026-10-22 14:24:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5857 [2026-10-22 14:24:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7115 [2026-10-22 14:25:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1726 [2026-10-22 14:25:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6341 [2026-10-22 14:25:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3982 [2026-10-22 14:25:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1190 [2026-10-22 14:25:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9155 [2026-10-22 14:25:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.1787 [2026-10-22 14:25:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4765 [2026-10-22 14:25:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3346 [2026-10-22 14:25:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6014 [2026-10-22 14:25:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7323 [2026-10-22 14:25:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8049 [2026-10-22 14:25:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6108 [2026-10-22 14:26:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0670 [2026-10-22 14:26:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9328 [2026-10-22 14:26:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0322 [2026-10-22 14:26:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3554 [2026-10-22 14:26:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4559 [2026-10-22 14:26:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6572 [2026-10-22 14:26:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5347 [2026-10-22 14:26:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4718 [2026-10-22 14:26:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8392 [2026-10-22 14:26:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2219 [2026-10-22 14:26:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0958 [2026-10-22 14:26:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1748 [2026-10-22 14:26:50][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,598 | MFU: 1.8% [2026-10-22 14:26:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9666 [2026-10-22 14:26:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=3.8097 [2026-10-22 14:26:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3447 [2026-10-22 14:26:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0428 [2026-10-22 14:26:54][INFO][training.pretrain_engine] [Ph2|Step 794] loss=4.7390 | ppl=114.32 | aux=0.02504 | lr=2.96e-04 | gnorm=0.516 | tok/s=29,354 | MFU=7.0% | step=142889ms | tokens=1.635B | mem=4.2/32.7GB | cost=$65.14/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 20:55 (126.5h) | Phase3: SKIPPED | SFT: Oct 28 01:39 (131.2h) | DPO (Final): Oct 28 02:13 (131.8h) [2026-10-22 14:26:54][INFO][training.pretrain_engine] Starting global step 795 (accumulating 32 micro-batches)... [2026-10-22 14:26:54][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 14:27:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4288 [2026-10-22 14:27:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7931 [2026-10-22 14:27:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4611 [2026-10-22 14:27:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0263 [2026-10-22 14:27:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8178 [2026-10-22 14:27:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8159 [2026-10-22 14:27:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1252 [2026-10-22 14:27:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0824 [2026-10-22 14:27:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7604 [2026-10-22 14:27:47][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4848 [2026-10-22 14:27:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8132 [2026-10-22 14:27:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6834 [2026-10-22 14:28:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1929 [2026-10-22 14:28:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1170 [2026-10-22 14:28:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5971 [2026-10-22 14:28:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0566 [2026-10-22 14:28:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5122 [2026-10-22 14:28:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.2495 [2026-10-22 14:28:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9163 [2026-10-22 14:28:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0316 [2026-10-22 14:28:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6586 [2026-10-22 14:28:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7959 [2026-10-22 14:28:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0622 [2026-10-22 14:28:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9184 [2026-10-22 14:28:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3868 [2026-10-22 14:28:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2412 [2026-10-22 14:28:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3031 [2026-10-22 14:28:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3869 [2026-10-22 14:29:12][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,610 | MFU: 1.8% [2026-10-22 14:29:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6073 [2026-10-22 14:29:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7188 [2026-10-22 14:29:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0385 [2026-10-22 14:29:17][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0362 [2026-10-22 14:29:17][INFO][training.pretrain_engine] [Ph2|Step 795] loss=4.7192 | ppl=112.07 | aux=0.02505 | lr=2.96e-04 | gnorm=0.637 | tok/s=29,353 | MFU=7.0% | step=142893ms | tokens=1.636B | mem=4.2/32.7GB | cost=$65.23/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 20:57 (126.5h) | Phase3: SKIPPED | SFT: Oct 28 01:41 (131.2h) | DPO (Final): Oct 28 02:15 (131.8h) [2026-10-22 14:29:17][INFO][training.pretrain_engine] Starting global step 796 (accumulating 32 micro-batches)... [2026-10-22 14:29:17][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 14:29:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.5031 [2026-10-22 14:29:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9293 [2026-10-22 14:29:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8804 [2026-10-22 14:29:35][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6957 [2026-10-22 14:29:52][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4383 [2026-10-22 14:29:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4820 [2026-10-22 14:29:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5291 [2026-10-22 14:29:53][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3895 [2026-10-22 14:30:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4835 [2026-10-22 14:30:10][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.8929 [2026-10-22 14:30:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3486 [2026-10-22 14:30:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2001 [2026-10-22 14:30:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7218 [2026-10-22 14:30:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9368 [2026-10-22 14:30:28][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7951 [2026-10-22 14:30:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2223 [2026-10-22 14:30:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5564 [2026-10-22 14:30:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3976 [2026-10-22 14:30:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5490 [2026-10-22 14:30:46][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9611 [2026-10-22 14:31:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4426 [2026-10-22 14:31:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.1927 [2026-10-22 14:31:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=3.9558 [2026-10-22 14:31:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=3.7809 [2026-10-22 14:31:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=3.8578 [2026-10-22 14:31:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=3.8486 [2026-10-22 14:31:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3422 [2026-10-22 14:31:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5997 [2026-10-22 14:31:35][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,589 | MFU: 1.8% [2026-10-22 14:31:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5294 [2026-10-22 14:31:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8754 [2026-10-22 14:31:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7995 [2026-10-22 14:31:40][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5861 [2026-10-22 14:31:40][INFO][training.pretrain_engine] [Ph2|Step 796] loss=4.7974 | ppl=121.19 | aux=0.02506 | lr=2.96e-04 | gnorm=0.836 | tok/s=29,352 | MFU=7.0% | step=142895ms | tokens=1.637B | mem=4.2/32.7GB | cost=$65.31/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 20:59 (126.5h) | Phase3: SKIPPED | SFT: Oct 28 01:43 (131.2h) | DPO (Final): Oct 28 02:17 (131.8h) [2026-10-22 14:31:40][INFO][training.pretrain_engine] Starting global step 797 (accumulating 32 micro-batches)... [2026-10-22 14:31:40][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 14:31:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6998 [2026-10-22 14:31:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3893 [2026-10-22 14:31:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2260 [2026-10-22 14:31:58][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4173 [2026-10-22 14:32:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9448 [2026-10-22 14:32:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8781 [2026-10-22 14:32:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0851 [2026-10-22 14:32:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8951 [2026-10-22 14:32:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.1236 [2026-10-22 14:32:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6570 [2026-10-22 14:32:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2315 [2026-10-22 14:32:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2064 [2026-10-22 14:32:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1029 [2026-10-22 14:32:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0955 [2026-10-22 14:32:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9657 [2026-10-22 14:32:51][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5093 [2026-10-22 14:33:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=3.8704 [2026-10-22 14:33:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=3.3844 [2026-10-22 14:33:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5666 [2026-10-22 14:33:09][INFO][training.pretrain_engine] [accum 20/32] micro-loss=3.8994 [2026-10-22 14:33:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6124 [2026-10-22 14:33:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6657 [2026-10-22 14:33:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8479 [2026-10-22 14:33:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7165 [2026-10-22 14:33:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1611 [2026-10-22 14:33:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7835 [2026-10-22 14:33:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1116 [2026-10-22 14:33:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9163 [2026-10-22 14:33:58][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,624 | MFU: 1.8% [2026-10-22 14:34:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5938 [2026-10-22 14:34:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6988 [2026-10-22 14:34:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9322 [2026-10-22 14:34:03][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.1897 [2026-10-22 14:34:03][INFO][training.pretrain_engine] [Ph2|Step 797] loss=4.7136 | ppl=111.46 | aux=0.02506 | lr=2.96e-04 | gnorm=1.273 | tok/s=29,353 | MFU=7.0% | step=142890ms | tokens=1.638B | mem=4.2/32.7GB | cost=$65.39/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 21:00 (126.4h) | Phase3: SKIPPED | SFT: Oct 28 01:44 (131.2h) | DPO (Final): Oct 28 02:18 (131.7h) [2026-10-22 14:34:03][INFO][training.pretrain_engine] Starting global step 798 (accumulating 32 micro-batches)... [2026-10-22 14:34:03][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 14:34:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.7660 [2026-10-22 14:34:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3103 [2026-10-22 14:34:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.7965 [2026-10-22 14:34:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.1177 [2026-10-22 14:34:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2972 [2026-10-22 14:34:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4843 [2026-10-22 14:34:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4485 [2026-10-22 14:34:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1546 [2026-10-22 14:34:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8274 [2026-10-22 14:34:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8992 [2026-10-22 14:34:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8289 [2026-10-22 14:34:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9698 [2026-10-22 14:35:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5641 [2026-10-22 14:35:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7039 [2026-10-22 14:35:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3655 [2026-10-22 14:35:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8835 [2026-10-22 14:35:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6073 [2026-10-22 14:35:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=3.9249 [2026-10-22 14:35:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8249 [2026-10-22 14:35:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8506 [2026-10-22 14:35:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8595 [2026-10-22 14:35:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8615 [2026-10-22 14:35:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2221 [2026-10-22 14:35:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9002 [2026-10-22 14:36:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4109 [2026-10-22 14:36:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4111 [2026-10-22 14:36:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3369 [2026-10-22 14:36:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5642 [2026-10-22 14:36:20][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,624 | MFU: 1.8% [2026-10-22 14:36:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0964 [2026-10-22 14:36:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8475 [2026-10-22 14:36:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9469 [2026-10-22 14:36:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8753 [2026-10-22 14:36:25][INFO][training.pretrain_engine] [Ph2|Step 798] loss=4.7352 | ppl=113.89 | aux=0.02505 | lr=2.96e-04 | gnorm=0.922 | tok/s=29,354 | MFU=7.0% | step=142887ms | tokens=1.639B | mem=4.2/32.7GB | cost=$65.47/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 21:02 (126.4h) | Phase3: SKIPPED | SFT: Oct 28 01:46 (131.2h) | DPO (Final): Oct 28 02:20 (131.7h) [2026-10-22 14:36:25][INFO][training.pretrain_engine] Starting global step 799 (accumulating 32 micro-batches)... [2026-10-22 14:36:25][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 14:36:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1149 [2026-10-22 14:36:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6541 [2026-10-22 14:36:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.4251 [2026-10-22 14:36:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7676 [2026-10-22 14:37:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4018 [2026-10-22 14:37:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7019 [2026-10-22 14:37:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6753 [2026-10-22 14:37:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6447 [2026-10-22 14:37:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7528 [2026-10-22 14:37:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8072 [2026-10-22 14:37:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4720 [2026-10-22 14:37:18][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9761 [2026-10-22 14:37:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2311 [2026-10-22 14:37:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2901 [2026-10-22 14:37:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4332 [2026-10-22 14:37:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6271 [2026-10-22 14:37:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8476 [2026-10-22 14:37:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7889 [2026-10-22 14:37:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6556 [2026-10-22 14:37:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5341 [2026-10-22 14:38:11][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5262 [2026-10-22 14:38:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=3.5045 [2026-10-22 14:38:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.0536 [2026-10-22 14:38:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.0283 [2026-10-22 14:38:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.1480 [2026-10-22 14:38:29][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3719 [2026-10-22 14:38:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=3.9695 [2026-10-22 14:38:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7189 [2026-10-22 14:38:43][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,615 | MFU: 1.8% [2026-10-22 14:38:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7167 [2026-10-22 14:38:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8541 [2026-10-22 14:38:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7017 [2026-10-22 14:38:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0995 [2026-10-22 14:38:48][INFO][training.pretrain_engine] [Ph2|Step 799] loss=4.6043 | ppl=99.91 | aux=0.02505 | lr=2.96e-04 | gnorm=0.832 | tok/s=29,354 | MFU=7.0% | step=142888ms | tokens=1.640B | mem=4.2/32.7GB | cost=$65.55/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 21:04 (126.4h) | Phase3: SKIPPED | SFT: Oct 28 01:48 (131.2h) | DPO (Final): Oct 28 02:22 (131.7h) [2026-10-22 14:38:48][INFO][training.pretrain_engine] Starting global step 800 (accumulating 32 micro-batches)... [2026-10-22 14:38:48][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 14:39:05][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9170 [2026-10-22 14:39:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3772 [2026-10-22 14:39:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3657 [2026-10-22 14:39:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2251 [2026-10-22 14:39:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6559 [2026-10-22 14:39:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4223 [2026-10-22 14:39:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5941 [2026-10-22 14:39:23][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7371 [2026-10-22 14:39:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1947 [2026-10-22 14:39:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0064 [2026-10-22 14:39:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.3841 [2026-10-22 14:39:41][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8911 [2026-10-22 14:39:58][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.6397 [2026-10-22 14:39:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9880 [2026-10-22 14:39:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9666 [2026-10-22 14:39:59][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6133 [2026-10-22 14:40:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9510 [2026-10-22 14:40:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6609 [2026-10-22 14:40:16][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8895 [2026-10-22 14:40:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9108 [2026-10-22 14:40:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7302 [2026-10-22 14:40:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9572 [2026-10-22 14:40:34][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6514 [2026-10-22 14:40:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1045 [2026-10-22 14:40:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0245 [2026-10-22 14:40:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0744 [2026-10-22 14:40:52][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0956 [2026-10-22 14:40:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9388 [2026-10-22 14:41:05][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,630 | MFU: 1.8% [2026-10-22 14:41:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7062 [2026-10-22 14:41:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4829 [2026-10-22 14:41:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7008 [2026-10-22 14:41:11][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7634 [2026-10-22 14:41:11][INFO][training.pretrain_engine] [Ph2|Step 800] loss=4.7513 | ppl=115.74 | aux=0.02506 | lr=2.96e-04 | gnorm=0.906 | tok/s=29,353 | MFU=7.0% | step=142891ms | tokens=1.641B | mem=4.2/32.7GB | cost=$65.63/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 21:06 (126.4h) | Phase3: SKIPPED | SFT: Oct 28 01:50 (131.2h) | DPO (Final): Oct 28 02:24 (131.7h) [2026-10-22 14:41:11][INFO][training.pretrain_engine] Starting global step 801 (accumulating 32 micro-batches)... [2026-10-22 14:41:11][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 14:41:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9482 [2026-10-22 14:41:28][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7327 [2026-10-22 14:41:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2692 [2026-10-22 14:41:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2529 [2026-10-22 14:41:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5137 [2026-10-22 14:41:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1907 [2026-10-22 14:41:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6508 [2026-10-22 14:41:46][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1435 [2026-10-22 14:42:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=3.8512 [2026-10-22 14:42:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3792 [2026-10-22 14:42:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=3.8707 [2026-10-22 14:42:04][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4306 [2026-10-22 14:42:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=3.0751 [2026-10-22 14:42:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3262 [2026-10-22 14:42:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.0222 [2026-10-22 14:42:22][INFO][training.pretrain_engine] [accum 16/32] micro-loss=3.5464 [2026-10-22 14:42:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6268 [2026-10-22 14:42:39][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6116 [2026-10-22 14:42:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7051 [2026-10-22 14:42:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6574 [2026-10-22 14:42:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8167 [2026-10-22 14:42:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0353 [2026-10-22 14:42:57][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4589 [2026-10-22 14:42:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8523 [2026-10-22 14:43:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7293 [2026-10-22 14:43:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7281 [2026-10-22 14:43:15][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8044 [2026-10-22 14:43:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2390 [2026-10-22 14:43:29][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,602 | MFU: 1.8% [2026-10-22 14:43:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6059 [2026-10-22 14:43:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8364 [2026-10-22 14:43:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6661 [2026-10-22 14:43:34][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9129 [2026-10-22 14:43:34][INFO][training.pretrain_engine] [Ph2|Step 801] loss=4.7357 | ppl=113.95 | aux=0.02507 | lr=2.96e-04 | gnorm=1.031 | tok/s=29,352 | MFU=7.0% | step=142897ms | tokens=1.642B | mem=4.2/32.7GB | cost=$65.72/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 21:08 (126.4h) | Phase3: SKIPPED | SFT: Oct 28 01:52 (131.1h) | DPO (Final): Oct 28 02:26 (131.7h) [2026-10-22 14:43:37][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step801 | Tokens: 1.642B | Spent: $65.72 | Remaining: $934.28 [2026-10-22 14:43:37][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step801 [2026-10-22 14:43:37][INFO][training.pretrain_engine] Starting global step 802 (accumulating 32 micro-batches)... [2026-10-22 14:43:37][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 14:43:37][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-22 14:43:37][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-22 14:43:37][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-22 14:43:37][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step801 to Phase-Technologies/forge-500m-pretrain (path: phase2_step801)... [2026-10-22 14:43:37][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-22 14:43:46][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-22 14:43:46][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step801 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-22 14:43:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0389 [2026-10-22 14:43:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3808 [2026-10-22 14:43:52][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6948 [2026-10-22 14:43:55][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9083 [2026-10-22 14:44:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6399 [2026-10-22 14:44:09][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2880 [2026-10-22 14:44:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5760 [2026-10-22 14:44:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2691 [2026-10-22 14:44:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1054 [2026-10-22 14:44:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0478 [2026-10-22 14:44:27][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6727 [2026-10-22 14:44:30][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1762 [2026-10-22 14:44:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3284 [2026-10-22 14:44:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4554 [2026-10-22 14:44:45][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6477 [2026-10-22 14:44:48][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3014 [2026-10-22 14:45:02][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8608 [2026-10-22 14:45:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4538 [2026-10-22 14:45:03][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6663 [2026-10-22 14:45:06][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8323 [2026-10-22 14:45:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7058 [2026-10-22 14:45:20][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8128 [2026-10-22 14:45:21][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8033 [2026-10-22 14:45:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2435 [2026-10-22 14:45:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8306 [2026-10-22 14:45:38][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8291 [2026-10-22 14:45:39][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4538 [2026-10-22 14:45:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7686 [2026-10-22 14:45:55][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,605 | MFU: 1.8% [2026-10-22 14:45:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8289 [2026-10-22 14:45:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8018 [2026-10-22 14:45:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8459 [2026-10-22 14:45:59][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9145 [2026-10-22 14:45:59][INFO][training.pretrain_engine] [Ph2|Step 802] loss=4.5980 | ppl=99.29 | aux=0.02506 | lr=2.96e-04 | gnorm=0.914 | tok/s=29,353 | MFU=7.0% | step=142893ms | tokens=1.643B | mem=4.2/32.7GB | cost=$65.80/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 21:10 (126.4h) | Phase3: SKIPPED | SFT: Oct 28 01:54 (131.1h) | DPO (Final): Oct 28 02:28 (131.7h) [2026-10-22 14:45:59][INFO][training.pretrain_engine] Starting global step 803 (accumulating 32 micro-batches)... [2026-10-22 14:45:59][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 14:46:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6618 [2026-10-22 14:46:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3512 [2026-10-22 14:46:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5182 [2026-10-22 14:46:17][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3718 [2026-10-22 14:46:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5708 [2026-10-22 14:46:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.1909 [2026-10-22 14:46:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2523 [2026-10-22 14:46:35][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6834 [2026-10-22 14:46:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2686 [2026-10-22 14:46:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5091 [2026-10-22 14:46:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2078 [2026-10-22 14:46:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2570 [2026-10-22 14:47:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0057 [2026-10-22 14:47:10][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5957 [2026-10-22 14:47:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6966 [2026-10-22 14:47:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8635 [2026-10-22 14:47:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5866 [2026-10-22 14:47:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9904 [2026-10-22 14:47:28][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7044 [2026-10-22 14:47:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.0098 [2026-10-22 14:47:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2780 [2026-10-22 14:47:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=3.9742 [2026-10-22 14:47:46][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7627 [2026-10-22 14:47:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4860 [2026-10-22 14:48:03][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2666 [2026-10-22 14:48:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3734 [2026-10-22 14:48:04][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9774 [2026-10-22 14:48:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5665 [2026-10-22 14:48:17][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,625 | MFU: 1.8% [2026-10-22 14:48:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6247 [2026-10-22 14:48:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7387 [2026-10-22 14:48:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6215 [2026-10-22 14:48:22][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0406 [2026-10-22 14:48:22][INFO][training.pretrain_engine] [Ph2|Step 803] loss=4.6210 | ppl=101.59 | aux=0.02505 | lr=2.96e-04 | gnorm=0.863 | tok/s=29,352 | MFU=7.0% | step=142896ms | tokens=1.644B | mem=4.2/32.7GB | cost=$65.88/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 21:12 (126.4h) | Phase3: SKIPPED | SFT: Oct 28 01:55 (131.1h) | DPO (Final): Oct 28 02:30 (131.7h) [2026-10-22 14:48:22][INFO][training.pretrain_engine] Starting global step 804 (accumulating 32 micro-batches)... [2026-10-22 14:48:22][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 14:48:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8994 [2026-10-22 14:48:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2815 [2026-10-22 14:48:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4652 [2026-10-22 14:48:40][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6446 [2026-10-22 14:48:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1333 [2026-10-22 14:48:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2589 [2026-10-22 14:48:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.3871 [2026-10-22 14:48:58][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1821 [2026-10-22 14:49:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3986 [2026-10-22 14:49:15][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8492 [2026-10-22 14:49:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.0094 [2026-10-22 14:49:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.0962 [2026-10-22 14:49:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=3.5857 [2026-10-22 14:49:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3257 [2026-10-22 14:49:33][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5114 [2026-10-22 14:49:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8211 [2026-10-22 14:49:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3918 [2026-10-22 14:49:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6558 [2026-10-22 14:49:51][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.1679 [2026-10-22 14:49:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4993 [2026-10-22 14:50:08][INFO][training.pretrain_engine] [accum 24/32] micro-loss=3.9455 [2026-10-22 14:50:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2961 [2026-10-22 14:50:09][INFO][training.pretrain_engine] [accum 24/32] micro-loss=3.8567 [2026-10-22 14:50:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4414 [2026-10-22 14:50:26][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3086 [2026-10-22 14:50:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.1651 [2026-10-22 14:50:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2078 [2026-10-22 14:50:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7037 [2026-10-22 14:50:40][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,611 | MFU: 1.8% [2026-10-22 14:50:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5871 [2026-10-22 14:50:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0531 [2026-10-22 14:50:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2631 [2026-10-22 14:50:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1360 [2026-10-22 14:50:46][INFO][training.pretrain_engine] [Ph2|Step 804] loss=4.6820 | ppl=107.98 | aux=0.02505 | lr=2.96e-04 | gnorm=0.879 | tok/s=29,352 | MFU=7.0% | step=142898ms | tokens=1.645B | mem=4.2/32.7GB | cost=$65.96/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 21:13 (126.4h) | Phase3: SKIPPED | SFT: Oct 28 01:57 (131.1h) | DPO (Final): Oct 28 02:31 (131.7h) [2026-10-22 14:50:46][INFO][training.pretrain_engine] Starting global step 805 (accumulating 32 micro-batches)... [2026-10-22 14:50:46][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 14:51:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.9569 [2026-10-22 14:51:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.3483 [2026-10-22 14:51:03][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.1326 [2026-10-22 14:51:04][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.0228 [2026-10-22 14:51:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5814 [2026-10-22 14:51:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5039 [2026-10-22 14:51:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4336 [2026-10-22 14:51:21][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7797 [2026-10-22 14:51:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=3.9015 [2026-10-22 14:51:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4327 [2026-10-22 14:51:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=3.5612 [2026-10-22 14:51:39][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4628 [2026-10-22 14:51:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3983 [2026-10-22 14:51:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2517 [2026-10-22 14:51:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3774 [2026-10-22 14:51:57][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5447 [2026-10-22 14:52:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7752 [2026-10-22 14:52:14][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8700 [2026-10-22 14:52:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8073 [2026-10-22 14:52:15][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7355 [2026-10-22 14:52:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4518 [2026-10-22 14:52:32][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3922 [2026-10-22 14:52:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8897 [2026-10-22 14:52:33][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4305 [2026-10-22 14:52:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0794 [2026-10-22 14:52:50][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8316 [2026-10-22 14:52:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6546 [2026-10-22 14:52:51][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4417 [2026-10-22 14:53:03][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,611 | MFU: 1.8% [2026-10-22 14:53:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4602 [2026-10-22 14:53:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3371 [2026-10-22 14:53:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6222 [2026-10-22 14:53:09][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4401 [2026-10-22 14:53:09][INFO][training.pretrain_engine] [Ph2|Step 805] loss=4.5855 | ppl=98.05 | aux=0.02506 | lr=2.96e-04 | gnorm=0.676 | tok/s=29,352 | MFU=7.0% | step=142896ms | tokens=1.646B | mem=4.2/32.7GB | cost=$66.04/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 21:15 (126.4h) | Phase3: SKIPPED | SFT: Oct 28 01:59 (131.1h) | DPO (Final): Oct 28 02:33 (131.7h) [2026-10-22 14:53:09][INFO][training.pretrain_engine] Starting global step 806 (accumulating 32 micro-batches)... [2026-10-22 14:53:09][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 14:53:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9001 [2026-10-22 14:53:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.9391 [2026-10-22 14:53:26][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2938 [2026-10-22 14:53:27][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1168 [2026-10-22 14:53:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8862 [2026-10-22 14:53:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7469 [2026-10-22 14:53:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8806 [2026-10-22 14:53:44][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6989 [2026-10-22 14:54:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6751 [2026-10-22 14:54:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8378 [2026-10-22 14:54:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5134 [2026-10-22 14:54:02][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7132 [2026-10-22 14:54:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7048 [2026-10-22 14:54:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1083 [2026-10-22 14:54:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8659 [2026-10-22 14:54:20][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0074 [2026-10-22 14:54:37][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5259 [2026-10-22 14:54:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7928 [2026-10-22 14:54:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6594 [2026-10-22 14:54:38][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7428 [2026-10-22 14:54:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9508 [2026-10-22 14:54:55][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8477 [2026-10-22 14:54:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6984 [2026-10-22 14:54:56][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7577 [2026-10-22 14:55:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.0267 [2026-10-22 14:55:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8527 [2026-10-22 14:55:13][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1729 [2026-10-22 14:55:14][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2062 [2026-10-22 14:55:27][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,591 | MFU: 1.8% [2026-10-22 14:55:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1750 [2026-10-22 14:55:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1437 [2026-10-22 14:55:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8841 [2026-10-22 14:55:31][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2995 [2026-10-22 14:55:31][INFO][training.pretrain_engine] [Ph2|Step 806] loss=4.6962 | ppl=109.53 | aux=0.02505 | lr=2.96e-04 | gnorm=0.973 | tok/s=29,353 | MFU=7.0% | step=142892ms | tokens=1.647B | mem=4.2/32.7GB | cost=$66.13/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 21:17 (126.4h) | Phase3: SKIPPED | SFT: Oct 28 02:01 (131.1h) | DPO (Final): Oct 28 02:35 (131.7h) [2026-10-22 14:55:31][INFO][training.pretrain_engine] Starting global step 807 (accumulating 32 micro-batches)... [2026-10-22 14:55:31][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 14:55:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.9338 [2026-10-22 14:55:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6812 [2026-10-22 14:55:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.8426 [2026-10-22 14:55:49][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6112 [2026-10-22 14:56:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8051 [2026-10-22 14:56:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8688 [2026-10-22 14:56:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.4754 [2026-10-22 14:56:07][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8520 [2026-10-22 14:56:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6106 [2026-10-22 14:56:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7495 [2026-10-22 14:56:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8790 [2026-10-22 14:56:25][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9551 [2026-10-22 14:56:42][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8618 [2026-10-22 14:56:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7859 [2026-10-22 14:56:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8086 [2026-10-22 14:56:43][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5784 [2026-10-22 14:57:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.1447 [2026-10-22 14:57:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9364 [2026-10-22 14:57:00][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7160 [2026-10-22 14:57:01][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5535 [2026-10-22 14:57:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.5228 [2026-10-22 14:57:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7051 [2026-10-22 14:57:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2117 [2026-10-22 14:57:18][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9939 [2026-10-22 14:57:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6818 [2026-10-22 14:57:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6564 [2026-10-22 14:57:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2841 [2026-10-22 14:57:36][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0357 [2026-10-22 14:57:49][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,613 | MFU: 1.8% [2026-10-22 14:57:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7337 [2026-10-22 14:57:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6755 [2026-10-22 14:57:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7934 [2026-10-22 14:57:54][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5192 [2026-10-22 14:57:54][INFO][training.pretrain_engine] [Ph2|Step 807] loss=4.7843 | ppl=119.62 | aux=0.02505 | lr=2.96e-04 | gnorm=0.824 | tok/s=29,354 | MFU=7.0% | step=142889ms | tokens=1.648B | mem=4.2/32.7GB | cost=$66.21/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 21:18 (126.3h) | Phase3: SKIPPED | SFT: Oct 28 02:02 (131.1h) | DPO (Final): Oct 28 02:36 (131.6h) [2026-10-22 14:57:54][INFO][training.pretrain_engine] Starting global step 808 (accumulating 32 micro-batches)... [2026-10-22 14:57:54][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 14:58:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6579 [2026-10-22 14:58:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1245 [2026-10-22 14:58:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0463 [2026-10-22 14:58:12][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1683 [2026-10-22 14:58:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7712 [2026-10-22 14:58:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7079 [2026-10-22 14:58:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.0732 [2026-10-22 14:58:30][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3817 [2026-10-22 14:58:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1371 [2026-10-22 14:58:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9458 [2026-10-22 14:58:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9511 [2026-10-22 14:58:48][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9039 [2026-10-22 14:59:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1972 [2026-10-22 14:59:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0946 [2026-10-22 14:59:05][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8426 [2026-10-22 14:59:06][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5238 [2026-10-22 14:59:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4414 [2026-10-22 14:59:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.2416 [2026-10-22 14:59:23][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3168 [2026-10-22 14:59:24][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.1425 [2026-10-22 14:59:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9837 [2026-10-22 14:59:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2610 [2026-10-22 14:59:41][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1353 [2026-10-22 14:59:42][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1150 [2026-10-22 14:59:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8012 [2026-10-22 14:59:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.9816 [2026-10-22 14:59:59][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8140 [2026-10-22 15:00:00][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1409 [2026-10-22 15:00:12][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,608 | MFU: 1.8% [2026-10-22 15:00:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7030 [2026-10-22 15:00:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3969 [2026-10-22 15:00:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4268 [2026-10-22 15:00:18][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5662 [2026-10-22 15:00:18][INFO][training.pretrain_engine] [Ph2|Step 808] loss=4.7741 | ppl=118.41 | aux=0.02506 | lr=2.96e-04 | gnorm=0.723 | tok/s=29,353 | MFU=7.0% | step=142894ms | tokens=1.649B | mem=4.2/32.7GB | cost=$66.29/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 21:20 (126.3h) | Phase3: SKIPPED | SFT: Oct 28 02:04 (131.1h) | DPO (Final): Oct 28 02:38 (131.6h) [2026-10-22 15:00:18][INFO][training.pretrain_engine] Starting global step 809 (accumulating 32 micro-batches)... [2026-10-22 15:00:18][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 15:00:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.1000 [2026-10-22 15:00:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.0772 [2026-10-22 15:00:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.1958 [2026-10-22 15:00:36][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4389 [2026-10-22 15:00:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7822 [2026-10-22 15:00:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0769 [2026-10-22 15:00:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1216 [2026-10-22 15:00:54][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9409 [2026-10-22 15:01:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0670 [2026-10-22 15:01:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6887 [2026-10-22 15:01:11][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9931 [2026-10-22 15:01:12][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.4238 [2026-10-22 15:01:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4297 [2026-10-22 15:01:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0080 [2026-10-22 15:01:29][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8241 [2026-10-22 15:01:30][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0037 [2026-10-22 15:01:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4688 [2026-10-22 15:01:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.2830 [2026-10-22 15:01:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.1881 [2026-10-22 15:01:47][INFO][training.pretrain_engine] [accum 20/32] micro-loss=3.8001 [2026-10-22 15:02:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2482 [2026-10-22 15:02:04][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8374 [2026-10-22 15:02:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5774 [2026-10-22 15:02:05][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9249 [2026-10-22 15:02:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0723 [2026-10-22 15:02:22][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0744 [2026-10-22 15:02:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1934 [2026-10-22 15:02:23][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1879 [2026-10-22 15:02:36][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,613 | MFU: 1.8% [2026-10-22 15:02:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4309 [2026-10-22 15:02:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2249 [2026-10-22 15:02:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9541 [2026-10-22 15:02:41][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8063 [2026-10-22 15:02:41][INFO][training.pretrain_engine] [Ph2|Step 809] loss=4.6860 | ppl=108.42 | aux=0.02506 | lr=2.96e-04 | gnorm=0.828 | tok/s=29,353 | MFU=7.0% | step=142891ms | tokens=1.650B | mem=4.2/32.7GB | cost=$66.37/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 21:22 (126.3h) | Phase3: SKIPPED | SFT: Oct 28 02:06 (131.1h) | DPO (Final): Oct 28 02:40 (131.6h) [2026-10-22 15:02:41][INFO][training.pretrain_engine] Starting global step 810 (accumulating 32 micro-batches)... [2026-10-22 15:02:41][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 15:02:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9511 [2026-10-22 15:02:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9199 [2026-10-22 15:02:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0357 [2026-10-22 15:02:59][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7220 [2026-10-22 15:03:16][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4647 [2026-10-22 15:03:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7237 [2026-10-22 15:03:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8830 [2026-10-22 15:03:17][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5070 [2026-10-22 15:03:34][INFO][training.pretrain_engine] [accum 12/32] micro-loss=3.2878 [2026-10-22 15:03:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6101 [2026-10-22 15:03:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5577 [2026-10-22 15:03:35][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.3291 [2026-10-22 15:03:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8717 [2026-10-22 15:03:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3318 [2026-10-22 15:03:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7349 [2026-10-22 15:03:52][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.0781 [2026-10-22 15:04:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8332 [2026-10-22 15:04:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8475 [2026-10-22 15:04:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8711 [2026-10-22 15:04:10][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0349 [2026-10-22 15:04:27][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9092 [2026-10-22 15:04:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.6566 [2026-10-22 15:04:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2321 [2026-10-22 15:04:28][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3159 [2026-10-22 15:04:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.2858 [2026-10-22 15:04:45][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9840 [2026-10-22 15:04:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0412 [2026-10-22 15:04:46][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5790 [2026-10-22 15:04:59][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,589 | MFU: 1.8% [2026-10-22 15:05:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8013 [2026-10-22 15:05:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9256 [2026-10-22 15:05:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9999 [2026-10-22 15:05:04][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8988 [2026-10-22 15:05:04][INFO][training.pretrain_engine] [Ph2|Step 810] loss=4.7360 | ppl=113.98 | aux=0.02506 | lr=2.96e-04 | gnorm=0.633 | tok/s=29,353 | MFU=7.0% | step=142890ms | tokens=1.652B | mem=4.2/32.7GB | cost=$66.45/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 21:24 (126.3h) | Phase3: SKIPPED | SFT: Oct 28 02:08 (131.1h) | DPO (Final): Oct 28 02:42 (131.6h) [2026-10-22 15:05:07][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step810 | Tokens: 1.652B | Spent: $66.46 | Remaining: $933.54 [2026-10-22 15:05:07][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step810 [2026-10-22 15:05:07][INFO][training.pretrain_engine] Starting global step 811 (accumulating 32 micro-batches)... [2026-10-22 15:05:07][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 15:05:07][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-22 15:05:07][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-22 15:05:07][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-22 15:05:07][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step810 to Phase-Technologies/forge-500m-pretrain (path: phase2_step810)... [2026-10-22 15:05:07][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-22 15:05:18][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-22 15:05:18][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step810 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-22 15:05:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.9075 [2026-10-22 15:05:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.9764 [2026-10-22 15:05:22][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.1216 [2026-10-22 15:05:25][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3709 [2026-10-22 15:05:39][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3656 [2026-10-22 15:05:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.0876 [2026-10-22 15:05:40][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.0219 [2026-10-22 15:05:43][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.2248 [2026-10-22 15:05:57][INFO][training.pretrain_engine] [accum 12/32] micro-loss=3.8396 [2026-10-22 15:05:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2150 [2026-10-22 15:05:58][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.0796 [2026-10-22 15:06:00][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4173 [2026-10-22 15:06:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0009 [2026-10-22 15:06:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9401 [2026-10-22 15:06:15][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7223 [2026-10-22 15:06:18][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8789 [2026-10-22 15:06:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0793 [2026-10-22 15:06:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5418 [2026-10-22 15:06:33][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4544 [2026-10-22 15:06:36][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5821 [2026-10-22 15:06:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1131 [2026-10-22 15:06:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9218 [2026-10-22 15:06:51][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8438 [2026-10-22 15:06:54][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5397 [2026-10-22 15:07:08][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5617 [2026-10-22 15:07:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6767 [2026-10-22 15:07:09][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9760 [2026-10-22 15:07:12][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0004 [2026-10-22 15:07:25][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,595 | MFU: 1.8% [2026-10-22 15:07:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6131 [2026-10-22 15:07:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8012 [2026-10-22 15:07:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5565 [2026-10-22 15:07:29][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6395 [2026-10-22 15:07:29][INFO][training.pretrain_engine] [Ph2|Step 811] loss=4.7240 | ppl=112.62 | aux=0.02505 | lr=2.96e-04 | gnorm=0.637 | tok/s=29,354 | MFU=7.0% | step=142887ms | tokens=1.653B | mem=4.2/32.7GB | cost=$66.54/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 21:25 (126.3h) | Phase3: SKIPPED | SFT: Oct 28 02:09 (131.0h) | DPO (Final): Oct 28 02:43 (131.6h) [2026-10-22 15:07:29][INFO][training.pretrain_engine] Starting global step 812 (accumulating 32 micro-batches)... [2026-10-22 15:07:29][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 15:07:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=3.8585 [2026-10-22 15:07:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6201 [2026-10-22 15:07:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3001 [2026-10-22 15:07:47][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7167 [2026-10-22 15:08:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8206 [2026-10-22 15:08:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5894 [2026-10-22 15:08:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.9629 [2026-10-22 15:08:05][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.0425 [2026-10-22 15:08:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8914 [2026-10-22 15:08:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6767 [2026-10-22 15:08:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9167 [2026-10-22 15:08:23][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0960 [2026-10-22 15:08:40][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3378 [2026-10-22 15:08:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9268 [2026-10-22 15:08:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.2335 [2026-10-22 15:08:41][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0721 [2026-10-22 15:08:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9500 [2026-10-22 15:08:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0135 [2026-10-22 15:08:58][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9861 [2026-10-22 15:08:59][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.3318 [2026-10-22 15:09:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0586 [2026-10-22 15:09:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3505 [2026-10-22 15:09:16][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0904 [2026-10-22 15:09:17][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7828 [2026-10-22 15:09:33][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7800 [2026-10-22 15:09:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8534 [2026-10-22 15:09:34][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.2481 [2026-10-22 15:09:35][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9850 [2026-10-22 15:09:48][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,594 | MFU: 1.8% [2026-10-22 15:09:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7211 [2026-10-22 15:09:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1003 [2026-10-22 15:09:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.2230 [2026-10-22 15:09:53][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1323 [2026-10-22 15:09:53][INFO][training.pretrain_engine] [Ph2|Step 812] loss=4.7435 | ppl=114.83 | aux=0.02505 | lr=2.96e-04 | gnorm=0.586 | tok/s=29,352 | MFU=7.0% | step=142897ms | tokens=1.654B | mem=4.2/32.7GB | cost=$66.62/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 21:28 (126.3h) | Phase3: SKIPPED | SFT: Oct 28 02:12 (131.0h) | DPO (Final): Oct 28 02:46 (131.6h) [2026-10-22 15:09:53][INFO][training.pretrain_engine] Starting global step 813 (accumulating 32 micro-batches)... [2026-10-22 15:09:53][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 15:10:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8766 [2026-10-22 15:10:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.1707 [2026-10-22 15:10:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.0908 [2026-10-22 15:10:10][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5331 [2026-10-22 15:10:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0611 [2026-10-22 15:10:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8544 [2026-10-22 15:10:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.2108 [2026-10-22 15:10:28][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5224 [2026-10-22 15:10:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8494 [2026-10-22 15:10:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9426 [2026-10-22 15:10:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6656 [2026-10-22 15:10:46][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9377 [2026-10-22 15:11:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.3443 [2026-10-22 15:11:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8695 [2026-10-22 15:11:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5944 [2026-10-22 15:11:04][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8859 [2026-10-22 15:11:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7476 [2026-10-22 15:11:21][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.2661 [2026-10-22 15:11:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4287 [2026-10-22 15:11:22][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4028 [2026-10-22 15:11:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4765 [2026-10-22 15:11:39][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8725 [2026-10-22 15:11:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5481 [2026-10-22 15:11:40][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.2450 [2026-10-22 15:11:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7750 [2026-10-22 15:11:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6340 [2026-10-22 15:11:57][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3853 [2026-10-22 15:11:58][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.1381 [2026-10-22 15:12:11][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,585 | MFU: 1.8% [2026-10-22 15:12:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3612 [2026-10-22 15:12:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8772 [2026-10-22 15:12:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4898 [2026-10-22 15:12:16][INFO][training.pretrain_engine] [accum 32/32] micro-loss=3.9884 [2026-10-22 15:12:16][INFO][training.pretrain_engine] [Ph2|Step 813] loss=4.5919 | ppl=98.69 | aux=0.02506 | lr=2.96e-04 | gnorm=0.574 | tok/s=29,352 | MFU=7.0% | step=142898ms | tokens=1.655B | mem=4.2/32.7GB | cost=$66.70/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 21:30 (126.3h) | Phase3: SKIPPED | SFT: Oct 28 02:14 (131.0h) | DPO (Final): Oct 28 02:48 (131.6h) [2026-10-22 15:12:16][INFO][training.pretrain_engine] Starting global step 814 (accumulating 32 micro-batches)... [2026-10-22 15:12:16][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 15:12:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5940 [2026-10-22 15:12:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8595 [2026-10-22 15:12:33][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4730 [2026-10-22 15:12:34][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7229 [2026-10-22 15:12:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8740 [2026-10-22 15:12:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7244 [2026-10-22 15:12:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5433 [2026-10-22 15:12:51][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7170 [2026-10-22 15:13:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8914 [2026-10-22 15:13:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6298 [2026-10-22 15:13:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9151 [2026-10-22 15:13:09][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9182 [2026-10-22 15:13:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8876 [2026-10-22 15:13:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0015 [2026-10-22 15:13:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4783 [2026-10-22 15:13:27][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1223 [2026-10-22 15:13:44][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7485 [2026-10-22 15:13:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9870 [2026-10-22 15:13:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0258 [2026-10-22 15:13:45][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5088 [2026-10-22 15:14:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.3999 [2026-10-22 15:14:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0782 [2026-10-22 15:14:02][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2042 [2026-10-22 15:14:03][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0375 [2026-10-22 15:14:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=3.7578 [2026-10-22 15:14:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.1336 [2026-10-22 15:14:20][INFO][training.pretrain_engine] [accum 28/32] micro-loss=3.8507 [2026-10-22 15:14:21][INFO][training.pretrain_engine] [accum 28/32] micro-loss=3.6667 [2026-10-22 15:14:34][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,604 | MFU: 1.8% [2026-10-22 15:14:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7207 [2026-10-22 15:14:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0229 [2026-10-22 15:14:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7990 [2026-10-22 15:14:39][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5917 [2026-10-22 15:14:39][INFO][training.pretrain_engine] [Ph2|Step 814] loss=4.6382 | ppl=103.36 | aux=0.02506 | lr=2.96e-04 | gnorm=0.668 | tok/s=29,351 | MFU=7.0% | step=142903ms | tokens=1.656B | mem=4.2/32.7GB | cost=$66.78/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 21:32 (126.3h) | Phase3: SKIPPED | SFT: Oct 28 02:16 (131.0h) | DPO (Final): Oct 28 02:50 (131.6h) [2026-10-22 15:14:39][INFO][training.pretrain_engine] Starting global step 815 (accumulating 32 micro-batches)... [2026-10-22 15:14:39][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 15:14:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3153 [2026-10-22 15:14:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.2909 [2026-10-22 15:14:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0349 [2026-10-22 15:14:57][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9415 [2026-10-22 15:15:14][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8089 [2026-10-22 15:15:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8921 [2026-10-22 15:15:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5911 [2026-10-22 15:15:15][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3888 [2026-10-22 15:15:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8624 [2026-10-22 15:15:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6953 [2026-10-22 15:15:32][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8629 [2026-10-22 15:15:33][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5555 [2026-10-22 15:15:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.1033 [2026-10-22 15:15:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7095 [2026-10-22 15:15:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6886 [2026-10-22 15:15:50][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7671 [2026-10-22 15:16:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=3.8339 [2026-10-22 15:16:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=3.7034 [2026-10-22 15:16:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=3.4146 [2026-10-22 15:16:08][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4002 [2026-10-22 15:16:25][INFO][training.pretrain_engine] [accum 24/32] micro-loss=3.7941 [2026-10-22 15:16:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7648 [2026-10-22 15:16:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=3.8985 [2026-10-22 15:16:26][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5437 [2026-10-22 15:16:43][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7367 [2026-10-22 15:16:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5115 [2026-10-22 15:16:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=3.7368 [2026-10-22 15:16:44][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6448 [2026-10-22 15:16:57][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,593 | MFU: 1.8% [2026-10-22 15:17:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6078 [2026-10-22 15:17:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8415 [2026-10-22 15:17:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2291 [2026-10-22 15:17:02][INFO][training.pretrain_engine] [accum 32/32] micro-loss=3.9329 [2026-10-22 15:17:02][INFO][training.pretrain_engine] [Ph2|Step 815] loss=4.7215 | ppl=112.34 | aux=0.02505 | lr=2.96e-04 | gnorm=0.570 | tok/s=29,351 | MFU=7.0% | step=142902ms | tokens=1.657B | mem=4.2/32.7GB | cost=$66.86/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 21:33 (126.3h) | Phase3: SKIPPED | SFT: Oct 28 02:17 (131.0h) | DPO (Final): Oct 28 02:51 (131.6h) [2026-10-22 15:17:02][INFO][training.pretrain_engine] Starting global step 816 (accumulating 32 micro-batches)... [2026-10-22 15:17:02][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 15:17:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6953 [2026-10-22 15:17:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9021 [2026-10-22 15:17:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.3017 [2026-10-22 15:17:20][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9386 [2026-10-22 15:17:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9003 [2026-10-22 15:17:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.0359 [2026-10-22 15:17:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9419 [2026-10-22 15:17:38][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9653 [2026-10-22 15:17:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0303 [2026-10-22 15:17:55][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2448 [2026-10-22 15:17:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2156 [2026-10-22 15:17:56][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1042 [2026-10-22 15:18:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2966 [2026-10-22 15:18:13][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.0132 [2026-10-22 15:18:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.2211 [2026-10-22 15:18:14][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.1037 [2026-10-22 15:18:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.3171 [2026-10-22 15:18:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7492 [2026-10-22 15:18:31][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8302 [2026-10-22 15:18:32][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0242 [2026-10-22 15:18:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1344 [2026-10-22 15:18:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1946 [2026-10-22 15:18:49][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1635 [2026-10-22 15:18:50][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9535 [2026-10-22 15:19:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1085 [2026-10-22 15:19:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9232 [2026-10-22 15:19:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6685 [2026-10-22 15:19:07][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7033 [2026-10-22 15:19:20][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,609 | MFU: 1.8% [2026-10-22 15:19:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7708 [2026-10-22 15:19:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4667 [2026-10-22 15:19:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5371 [2026-10-22 15:19:25][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0206 [2026-10-22 15:19:25][INFO][training.pretrain_engine] [Ph2|Step 816] loss=4.6611 | ppl=105.75 | aux=0.02505 | lr=2.96e-04 | gnorm=0.535 | tok/s=29,350 | MFU=7.0% | step=142904ms | tokens=1.658B | mem=4.2/32.7GB | cost=$66.95/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 21:35 (126.3h) | Phase3: SKIPPED | SFT: Oct 28 02:19 (131.0h) | DPO (Final): Oct 28 02:53 (131.6h) [2026-10-22 15:19:25][INFO][training.pretrain_engine] Starting global step 817 (accumulating 32 micro-batches)... [2026-10-22 15:19:25][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 15:19:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6821 [2026-10-22 15:19:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7460 [2026-10-22 15:19:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3649 [2026-10-22 15:19:43][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7444 [2026-10-22 15:20:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5690 [2026-10-22 15:20:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7897 [2026-10-22 15:20:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6830 [2026-10-22 15:20:01][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7842 [2026-10-22 15:20:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4210 [2026-10-22 15:20:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4679 [2026-10-22 15:20:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.0592 [2026-10-22 15:20:19][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.2306 [2026-10-22 15:20:36][INFO][training.pretrain_engine] [accum 16/32] micro-loss=3.9050 [2026-10-22 15:20:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9938 [2026-10-22 15:20:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.5539 [2026-10-22 15:20:37][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4123 [2026-10-22 15:20:54][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6254 [2026-10-22 15:20:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4651 [2026-10-22 15:20:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6138 [2026-10-22 15:20:55][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4369 [2026-10-22 15:21:12][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0426 [2026-10-22 15:21:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.2827 [2026-10-22 15:21:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7947 [2026-10-22 15:21:13][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1199 [2026-10-22 15:21:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1695 [2026-10-22 15:21:30][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.9638 [2026-10-22 15:21:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7735 [2026-10-22 15:21:31][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5621 [2026-10-22 15:21:43][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,618 | MFU: 1.8% [2026-10-22 15:21:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4686 [2026-10-22 15:21:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.1138 [2026-10-22 15:21:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4605 [2026-10-22 15:21:48][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.0553 [2026-10-22 15:21:49][INFO][training.pretrain_engine] [Ph2|Step 817] loss=4.6473 | ppl=104.30 | aux=0.02505 | lr=2.96e-04 | gnorm=0.488 | tok/s=29,350 | MFU=7.0% | step=142906ms | tokens=1.659B | mem=4.2/32.7GB | cost=$67.03/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 21:37 (126.3h) | Phase3: SKIPPED | SFT: Oct 28 02:21 (131.0h) | DPO (Final): Oct 28 02:55 (131.6h) [2026-10-22 15:21:49][INFO][training.pretrain_engine] Starting global step 818 (accumulating 32 micro-batches)... [2026-10-22 15:21:49][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 15:22:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7820 [2026-10-22 15:22:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.3195 [2026-10-22 15:22:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.7571 [2026-10-22 15:22:06][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.2309 [2026-10-22 15:22:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7739 [2026-10-22 15:22:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8403 [2026-10-22 15:22:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7316 [2026-10-22 15:22:24][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.7814 [2026-10-22 15:22:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1959 [2026-10-22 15:22:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6749 [2026-10-22 15:22:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0487 [2026-10-22 15:22:42][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0535 [2026-10-22 15:23:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7656 [2026-10-22 15:23:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8133 [2026-10-22 15:23:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9394 [2026-10-22 15:23:00][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7672 [2026-10-22 15:23:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0061 [2026-10-22 15:23:17][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5416 [2026-10-22 15:23:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7626 [2026-10-22 15:23:18][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5503 [2026-10-22 15:23:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7005 [2026-10-22 15:23:35][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7699 [2026-10-22 15:23:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7300 [2026-10-22 15:23:36][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7500 [2026-10-22 15:23:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3000 [2026-10-22 15:23:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8495 [2026-10-22 15:23:53][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.1520 [2026-10-22 15:23:54][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8383 [2026-10-22 15:24:06][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,611 | MFU: 1.8% [2026-10-22 15:24:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8896 [2026-10-22 15:24:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8098 [2026-10-22 15:24:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7192 [2026-10-22 15:24:12][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7695 [2026-10-22 15:24:12][INFO][training.pretrain_engine] [Ph2|Step 818] loss=4.7968 | ppl=121.12 | aux=0.02505 | lr=2.96e-04 | gnorm=0.570 | tok/s=29,350 | MFU=7.0% | step=142906ms | tokens=1.660B | mem=4.2/32.7GB | cost=$67.11/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 21:39 (126.3h) | Phase3: SKIPPED | SFT: Oct 28 02:23 (131.0h) | DPO (Final): Oct 28 02:57 (131.6h) [2026-10-22 15:24:12][INFO][training.pretrain_engine] Starting global step 819 (accumulating 32 micro-batches)... [2026-10-22 15:24:12][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 15:24:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6758 [2026-10-22 15:24:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6468 [2026-10-22 15:24:29][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5575 [2026-10-22 15:24:30][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.4210 [2026-10-22 15:24:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9659 [2026-10-22 15:24:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=5.1324 [2026-10-22 15:24:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.9082 [2026-10-22 15:24:47][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5375 [2026-10-22 15:25:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.0573 [2026-10-22 15:25:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8375 [2026-10-22 15:25:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4114 [2026-10-22 15:25:05][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7593 [2026-10-22 15:25:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7217 [2026-10-22 15:25:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8463 [2026-10-22 15:25:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8596 [2026-10-22 15:25:23][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.0031 [2026-10-22 15:25:40][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.0362 [2026-10-22 15:25:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.5732 [2026-10-22 15:25:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6120 [2026-10-22 15:25:41][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6234 [2026-10-22 15:25:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.0758 [2026-10-22 15:25:58][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8580 [2026-10-22 15:25:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.1105 [2026-10-22 15:25:59][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9844 [2026-10-22 15:26:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5380 [2026-10-22 15:26:16][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.3867 [2026-10-22 15:26:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0411 [2026-10-22 15:26:17][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6395 [2026-10-22 15:26:30][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,601 | MFU: 1.8% [2026-10-22 15:26:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6852 [2026-10-22 15:26:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5928 [2026-10-22 15:26:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.4626 [2026-10-22 15:26:35][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0866 [2026-10-22 15:26:35][INFO][training.pretrain_engine] [Ph2|Step 819] loss=4.7360 | ppl=113.98 | aux=0.02506 | lr=2.96e-04 | gnorm=0.551 | tok/s=29,350 | MFU=7.0% | step=142905ms | tokens=1.661B | mem=4.2/32.7GB | cost=$67.19/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 21:41 (126.2h) | Phase3: SKIPPED | SFT: Oct 28 02:25 (131.0h) | DPO (Final): Oct 28 02:59 (131.5h) [2026-10-22 15:26:38][INFO][training.pretrain_engine] ✅ [CHECKPOINT SAVED] phase2_step819 | Tokens: 1.661B | Spent: $67.19 | Remaining: $932.81 [2026-10-22 15:26:38][INFO][training.hub_uploader] Started background Hugging Face upload for /workspace/checkpoints/forge_500m_pretrain/phase2_step819 [2026-10-22 15:26:38][INFO][training.pretrain_engine] Starting global step 820 (accumulating 32 micro-batches)... [2026-10-22 15:26:38][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 15:26:38][INFO][httpx2] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK" [2026-10-22 15:26:38][INFO][training.hub_uploader] Checking/Creating repository Phase-Technologies/forge-500m-pretrain... [2026-10-22 15:26:38][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict" [2026-10-22 15:26:38][INFO][training.hub_uploader] Uploading folder /workspace/checkpoints/forge_500m_pretrain/phase2_step819 to Phase-Technologies/forge-500m-pretrain (path: phase2_step819)... [2026-10-22 15:26:38][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/preupload/main "HTTP/1.1 200 OK" [2026-10-22 15:26:47][INFO][httpx2] HTTP Request: POST https://huggingface.co/api/models/Phase-Technologies/forge-500m-pretrain/commit/main "HTTP/1.1 200 OK" [2026-10-22 15:26:47][INFO][training.hub_uploader] Successfully uploaded /workspace/checkpoints/forge_500m_pretrain/phase2_step819 to HF Hub (Phase-Technologies/forge-500m-pretrain) [2026-10-22 15:26:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0920 [2026-10-22 15:26:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0511 [2026-10-22 15:26:53][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8712 [2026-10-22 15:26:56][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.1045 [2026-10-22 15:27:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5892 [2026-10-22 15:27:10][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.8561 [2026-10-22 15:27:11][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.9219 [2026-10-22 15:27:13][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7524 [2026-10-22 15:27:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4430 [2026-10-22 15:27:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6145 [2026-10-22 15:27:28][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.1347 [2026-10-22 15:27:31][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.5879 [2026-10-22 15:27:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9399 [2026-10-22 15:27:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9686 [2026-10-22 15:27:46][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.9307 [2026-10-22 15:27:49][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7855 [2026-10-22 15:28:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7285 [2026-10-22 15:28:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.8305 [2026-10-22 15:28:04][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6293 [2026-10-22 15:28:07][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7606 [2026-10-22 15:28:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.7239 [2026-10-22 15:28:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.3664 [2026-10-22 15:28:22][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8856 [2026-10-22 15:28:24][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5173 [2026-10-22 15:28:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.6834 [2026-10-22 15:28:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8964 [2026-10-22 15:28:40][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8182 [2026-10-22 15:28:42][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.4395 [2026-10-22 15:28:56][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,611 | MFU: 1.8% [2026-10-22 15:29:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.1096 [2026-10-22 15:29:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7928 [2026-10-22 15:29:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.7317 [2026-10-22 15:29:00][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8353 [2026-10-22 15:29:00][INFO][training.pretrain_engine] [Ph2|Step 820] loss=4.7121 | ppl=111.29 | aux=0.02505 | lr=2.96e-04 | gnorm=0.424 | tok/s=29,353 | MFU=7.0% | step=142894ms | tokens=1.662B | mem=4.2/32.7GB | cost=$67.28/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 21:42 (126.2h) | Phase3: SKIPPED | SFT: Oct 28 02:26 (131.0h) | DPO (Final): Oct 28 03:00 (131.5h) [2026-10-22 15:29:00][INFO][training.pretrain_engine] Starting global step 821 (accumulating 32 micro-batches)... [2026-10-22 15:29:00][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 15:29:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9506 [2026-10-22 15:29:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6051 [2026-10-22 15:29:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.6765 [2026-10-22 15:29:18][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.5642 [2026-10-22 15:29:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5775 [2026-10-22 15:29:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.5362 [2026-10-22 15:29:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4671 [2026-10-22 15:29:36][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.7021 [2026-10-22 15:29:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.9716 [2026-10-22 15:29:53][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.7833 [2026-10-22 15:29:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.4383 [2026-10-22 15:29:54][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.6933 [2026-10-22 15:30:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7771 [2026-10-22 15:30:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.4738 [2026-10-22 15:30:11][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8794 [2026-10-22 15:30:12][INFO][training.pretrain_engine] [accum 16/32] micro-loss=5.5503 [2026-10-22 15:30:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4531 [2026-10-22 15:30:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6781 [2026-10-22 15:30:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.7447 [2026-10-22 15:30:29][INFO][training.pretrain_engine] [accum 20/32] micro-loss=3.9940 [2026-10-22 15:30:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9873 [2026-10-22 15:30:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=5.4005 [2026-10-22 15:30:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.9240 [2026-10-22 15:30:47][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8723 [2026-10-22 15:31:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.8425 [2026-10-22 15:31:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7189 [2026-10-22 15:31:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.5166 [2026-10-22 15:31:05][INFO][training.pretrain_engine] [accum 28/32] micro-loss=4.7886 [2026-10-22 15:31:18][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,606 | MFU: 1.8% [2026-10-22 15:31:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.9340 [2026-10-22 15:31:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=5.0211 [2026-10-22 15:31:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.5361 [2026-10-22 15:31:23][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8838 [2026-10-22 15:31:23][INFO][training.pretrain_engine] [Ph2|Step 821] loss=4.7173 | ppl=111.87 | aux=0.02505 | lr=2.96e-04 | gnorm=0.555 | tok/s=29,352 | MFU=7.0% | step=142895ms | tokens=1.663B | mem=4.2/32.7GB | cost=$67.36/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 21:44 (126.2h) | Phase3: SKIPPED | SFT: Oct 28 02:28 (130.9h) | DPO (Final): Oct 28 03:02 (131.5h) [2026-10-22 15:31:23][INFO][training.pretrain_engine] Starting global step 822 (accumulating 32 micro-batches)... [2026-10-22 15:31:23][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass... [2026-10-22 15:31:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0618 [2026-10-22 15:31:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=5.0210 [2026-10-22 15:31:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.8202 [2026-10-22 15:31:41][INFO][training.pretrain_engine] [accum 4/32] micro-loss=4.9412 [2026-10-22 15:31:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.3735 [2026-10-22 15:31:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.6961 [2026-10-22 15:31:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=3.9672 [2026-10-22 15:31:59][INFO][training.pretrain_engine] [accum 8/32] micro-loss=4.4722 [2026-10-22 15:32:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.1468 [2026-10-22 15:32:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=5.2914 [2026-10-22 15:32:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.8593 [2026-10-22 15:32:16][INFO][training.pretrain_engine] [accum 12/32] micro-loss=4.1929 [2026-10-22 15:32:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.7781 [2026-10-22 15:32:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8340 [2026-10-22 15:32:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.6983 [2026-10-22 15:32:34][INFO][training.pretrain_engine] [accum 16/32] micro-loss=4.8672 [2026-10-22 15:32:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=5.2893 [2026-10-22 15:32:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.6970 [2026-10-22 15:32:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.4218 [2026-10-22 15:32:52][INFO][training.pretrain_engine] [accum 20/32] micro-loss=4.9514 [2026-10-22 15:33:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.0462 [2026-10-22 15:33:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.4771 [2026-10-22 15:33:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.8946 [2026-10-22 15:33:10][INFO][training.pretrain_engine] [accum 24/32] micro-loss=4.5799 [2026-10-22 15:33:27][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.3675 [2026-10-22 15:33:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0000 [2026-10-22 15:33:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0269 [2026-10-22 15:33:28][INFO][training.pretrain_engine] [accum 28/32] micro-loss=5.0932 [2026-10-22 15:33:41][INFO][training.pretrain_engine] [Micro-step 31/32] Forward/backward... | Live Tok/s: 7,611 | MFU: 1.8% [2026-10-22 15:33:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.8119 [2026-10-22 15:33:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.6476 [2026-10-22 15:33:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.2268 [2026-10-22 15:33:46][INFO][training.pretrain_engine] [accum 32/32] micro-loss=4.3290 [2026-10-22 15:33:46][INFO][training.pretrain_engine] [Ph2|Step 822] loss=4.7022 | ppl=110.19 | aux=0.02505 | lr=2.96e-04 | gnorm=0.512 | tok/s=29,353 | MFU=7.0% | step=142890ms | tokens=1.664B | mem=4.2/32.7GB | cost=$67.44/$1000 ⏳ [STAGE ETAs] Phase1: SKIPPED | Phase2: Oct 27 21:45 (126.2h) | Phase3: SKIPPED | SFT: Oct 28 02:29 (130.9h) | DPO (Final): Oct 28 03:03 (131.5h) [2026-10-22 15:33:46][INFO][training.pretrain_engine] Starting global step 823 (accumulating 32 micro-batches)... [2026-10-22 15:33:46][INFO][training.pretrain_engine] [Micro-step 0/32] Forward/backward pass...